Sep 09 20:11:11.786041 ip-10-0-129-130 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:11:12.163192 ip-10-0-129-130 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:12.163192 ip-10-0-129-130 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:11:12.163192 ip-10-0-129-130 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:12.163192 ip-10-0-129-130 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:11:12.163192 ip-10-0-129-130 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:12.164702 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.164611    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:11:12.168115 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168100    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:12.168115 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168115    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168119    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168122    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168125    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168127    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168137    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168140    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168143    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168149    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168153    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168157    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168160    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168163    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168166    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168170    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168173    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168176    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168179    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168182    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:12.168179 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168185    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168188    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168191    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168194    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168197    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168200    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168202    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168205    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168207    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168210    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168212    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168215    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168218    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168220    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168223    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168227    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168231    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168233    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168236    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:12.168637 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168238    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168241    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168244    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168246    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168248    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168251    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168253    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168255    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168258    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168261    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168263    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168266    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168268    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168271    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168273    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168277    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168280    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168282    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168285    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168288    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:12.169163 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168290    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168292    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168295    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168297    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168300    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168303    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168305    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168308    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168310    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168313    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168315    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168317    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168321    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168323    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168326    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168328    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168331    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168334    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168336    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168338    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:12.169651 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168341    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168344    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168346    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168349    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168351    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168354    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168356    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168360    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168371    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168374    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168377    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168380    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168382    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168385    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168387    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168389    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168392    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168394    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168398    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.168400    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:12.170161 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169186    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169194    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169197    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169200    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169203    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169206    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169210    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169212    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169215    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169218    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169220    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169223    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169225    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169228    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169230    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169233    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169236    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169238    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169240    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169243    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:12.170644 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169246    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169249    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169251    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169254    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169258    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169261    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169264    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169267    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169270    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169272    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169275    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169278    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169281    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169283    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169286    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169288    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169291    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169294    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169296    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:12.171155 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169298    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169301    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169304    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169306    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169309    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169312    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169314    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169316    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169319    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169322    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169324    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169327    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169330    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169332    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169335    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169337    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169341    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169343    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169346    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169348    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:12.171615 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169350    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169353    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169355    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169358    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169360    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169364    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169368    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169371    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169373    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169376    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169379    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169381    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169384    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169387    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169390    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169395    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169399    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169402    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169405    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:12.172110 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169407    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169410    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169412    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169415    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169417    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169421    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169423    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169426    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169428    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169431    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169433    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169436    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169439    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169442    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169444    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169447    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169449    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169452    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169454    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169457    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:12.172673 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169460    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169871    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169877    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169881    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169884    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169887    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169890    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169893    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169895    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169898    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169901    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169903    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169905    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169908    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169910    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169913    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169916    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169919    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169922    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169925    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:12.173184 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169927    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169930    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169932    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169935    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169939    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169943    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169945    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169948    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169951    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169954    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169957    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169960    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169963    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169965    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169968    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169970    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169973    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169975    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169979    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169982    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:12.173688 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169985    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169987    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169990    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169992    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169995    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.169997    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170000    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170002    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170005    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170008    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170010    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170013    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170016    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170018    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170021    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170023    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170026    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170029    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170031    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:12.174173 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170034    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170036    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170038    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170041    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170044    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170046    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170049    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170052    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170056    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170059    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170061    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170064    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170066    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170069    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170071    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170074    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170076    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170079    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170081    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:12.174693 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170084    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170087    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170089    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170092    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170095    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170097    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170100    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170102    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170104    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170107    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170113    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170115    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170118    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170120    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170123    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170126    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170128    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170131    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170133    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170136    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:12.175164 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170139    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170141    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170203    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170234    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170241    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170245    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170249    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170252    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170256    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170260    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170263    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170266    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170270    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170273    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170277    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170280    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170283    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170286    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170289    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170292    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170296    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170299    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170303    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170306    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:11:12.175650 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170310    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170313    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170316    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170319    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170324    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170327    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170330    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170332    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170336    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170339    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170343    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170346    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170348    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170351    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170354    2563 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170357    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170361    2563 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170364    2563 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170366    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170369    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170372    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170376    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170378    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170382    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170385    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:11:12.176350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170388    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170391    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170394    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170396    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170399    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170403    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170406    2563 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170410    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170413    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170416    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170419    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170422    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170425    2563 flags.go:64] FLAG: --help="false"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170428    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170431    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170433    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170437    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170440    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170443    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170446    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170449    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170452    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170454    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:11:12.176992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170457    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170460    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170463    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170466    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170469    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170471    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170474    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170478    2563 flags.go:64] FLAG: --lock-file=""
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170481    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170484    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170487    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170492    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170495    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170498    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170502    2563 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170504    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170508    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170510    2563 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170513    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170517    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170520    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170524    2563 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170527    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170530    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170533    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:11:12.177727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170536    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170539    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170542    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170545    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170552    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170554    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170557    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170561    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170564    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170569    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170572    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170575    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170578    2563 flags.go:64] FLAG: --port="10250"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170581    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170584    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dffafb0d19671a3e"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170588    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170591    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170594    2563 flags.go:64] FLAG: --register-node="true"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170596    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170600    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170603    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170607    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170610    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170614    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170617    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:11:12.178829 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170620    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170622    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170625    2563 flags.go:64] FLAG: --runonce="false"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170628    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170631    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170634    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170636    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170639    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170642    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170646    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170649    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170652    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170655    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170658    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170660    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170663    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170666    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170669    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170674    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170677    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170679    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170684    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170688    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170690    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170693    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:11:12.179841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170696    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170699    2563 flags.go:64] FLAG: --v="2"
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170703    2563 flags.go:64] FLAG: --version="false"
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170712    2563 flags.go:64] FLAG: --vmodule=""
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170716    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.170719    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170852    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170857    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170860    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170863    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170866    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170868    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170871    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170873    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170878    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170881    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170884    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170888    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170891    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170894    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170896    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:12.180572 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170899    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170902    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170906    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170909    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170913    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170916    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170919    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170921    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170927    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170929    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170932    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170935    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170937    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170940    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170942    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170945    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170948    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170950    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170953    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170955    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:12.181197 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170958    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170961    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170963    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170966    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170968    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170971    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170973    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170976    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170978    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170982    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170985    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170987    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170990    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170993    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170995    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.170997    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171000    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171002    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171005    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171007    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:12.181696 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171012    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171015    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171018    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171020    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171023    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171025    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171028    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171030    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171033    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171035    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171038    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171041    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171043    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171045    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171048    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171050    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171053    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171056    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171058    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171061    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:12.182204 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171063    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171066    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171069    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171071    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171074    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171076    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171080    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171084    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171086    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171089    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171092    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171094    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171098    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171100    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171103    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171106    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171108    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171111    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171113    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:12.182701 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171116    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171119    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171122    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171124    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.171127    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.171675    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.180833    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.180850    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.181644    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.181652    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181710    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181716    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181720    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181724    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181727    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:12.183190 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181730    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181733    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181736    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181739    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181742    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181745    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181747    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181750    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181771    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181775    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181779    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181782    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181784    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181787    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181790    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181793    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181796    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181798    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181801    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:12.183583 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181804    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181808    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181812    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181815    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181819    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181823    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181826    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181828    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181831    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181834    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181837    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181840    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181842    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181845    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181849    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181852    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181855    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181857    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181860    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:12.184056 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181862    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181865    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181867    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181870    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181874    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181876    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181879    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181882    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181884    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181887    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181889    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181892    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181894    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181897    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181899    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181902    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181904    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181907    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181910    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:12.184574 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181913    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181916    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181918    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181921    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181923    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181926    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181929    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181931    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181934    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181937    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181939    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181942    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181944    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181947    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181950    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181952    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181955    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181958    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181961    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181963    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:12.185134 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181966    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181968    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181971    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181973    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181976    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181978    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181981    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181983    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181986    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181988    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181991    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181994    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181996    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.181999    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182003    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182007    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:12.185669 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182009    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.182015    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182136    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182141    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182144    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182147    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182150    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182152    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182155    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182158    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182160    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182163    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182165    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182168    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182171    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:12.186081 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182174    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182176    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182179    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182182    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182184    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182187    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182189    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182192    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182194    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182197    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182199    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182202    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182205    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182208    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182211    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182214    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182219    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182223    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182226    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182228    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:12.186474 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182231    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182233    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182236    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182238    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182242    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182244    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182246    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182249    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182251    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182254    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182257    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182259    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182264    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182267    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182270    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182272    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182275    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182278    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182281    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:12.186980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182283    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182286    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182289    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182291    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182294    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182296    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182299    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182302    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182304    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182307    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182309    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182312    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182314    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182317    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182319    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182322    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182324    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182326    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182329    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182331    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:12.187436 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182334    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182336    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182339    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182341    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182344    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182346    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182349    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182352    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182354    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182357    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182359    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182362    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182364    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182367    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182369    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182371    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182374    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182378    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182381    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:12.187956 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182384    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182387    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182390    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182393    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182395    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182400    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182403    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:12.182405    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.182411    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.182524    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.185203    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.186143    2563 server.go:1019] "Starting client certificate rotation"
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.186243    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:12.188435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.186276    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:12.188906 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.188509    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:12.192451 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.192433    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:12.206813 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.206793    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:11:12.210358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.210336    2563 log.go:25] "Validated CRI v1 image API"
Sep 09 20:11:12.211231 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.211214    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:11:12.213097 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.213079    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:12.215033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.215011    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80120494-dd90-4dec-a84f-0026ede5c718:/dev/nvme0n1p3 d9c6bfbf-bc41-4dd1-8e2f-c23f9a1aefcd:/dev/nvme0n1p4]
Sep 09 20:11:12.215117 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.215032    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:11:12.223179 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.223058    2563 manager.go:216] Machine: {Timestamp:2026-09-09 20:11:12.221566667 +0000 UTC m=+0.323627837 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100813 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28668f074fcc6edb44460afd3c4937 SystemUUID:ec28668f-074f-cc6e-db44-460afd3c4937 BootID:8bd8e8e4-95a8-4370-a0db-b11af8bcc120 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:26:4a:8c:51:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:26:4a:8c:51:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:09:f9:d3:b3:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:11:12.223179 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.223170    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:11:12.223312 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.223261    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:11:12.227507 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.227474    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:11:12.227647 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.227511    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:11:12.228488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.228478    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:11:12.228527 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.228489    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:11:12.228527 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.228498    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:12.229254 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.229243    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:12.230045 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.230036    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:11:12.232704 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.232693    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:12.232836 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.232827    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:11:12.232866 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.232857    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:11:12.232898 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.232867    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:11:12.232898 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.232879    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:11:12.232898 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.232888    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:11:12.232898 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.232897    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:11:12.233320 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.233304    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:12.233320 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.233329    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:12.236192 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.236177    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:11:12.237086 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.237070    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9j4cl"
Sep 09 20:11:12.239112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.239097    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:11:12.239164 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.239131    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:11:12.240294 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240279    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:11:12.240334 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240307    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:11:12.240334 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240319    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:11:12.240334 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240330    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240347    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240355    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240377    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240382    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240388    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240393    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240419    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:11:12.240423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.240424    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:11:12.242465 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.242445    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:11:12.242519 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.242441    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:11:12.242726 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.242715    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:11:12.242773 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.242728    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:11:12.243664 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.243648    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9j4cl"
Sep 09 20:11:12.246906 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.246892    2563 server.go:1268] "Started kubelet"
Sep 09 20:11:12.247002 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.246978    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:11:12.247054 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.246973    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:11:12.247099 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.247060    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:11:12.247681 ip-10-0-129-130 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:11:12.248645 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.248624    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:11:12.250008 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.249984    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:11:12.251369 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.251346    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:11:12.252645 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.252631    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:11:12.252711 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.252646    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:12.252842 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.252823    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.253279 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253256    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:11:12.253373 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253260    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:11:12.253373 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253313    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:11:12.253489 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253384    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:11:12.253489 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253394    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:11:12.253883 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253862    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:11:12.253972 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253890    2563 factory.go:55] Registering systemd factory
Sep 09 20:11:12.253972 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.253905    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:11:12.254172 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.254158    2563 factory.go:153] Registering CRI-O factory
Sep 09 20:11:12.254208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.254174    2563 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:11:12.254208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.254192    2563 factory.go:103] Registering Raw factory
Sep 09 20:11:12.254268 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.254218    2563 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:11:12.255122 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.255106    2563 manager.go:349] Starting recovery of all containers
Sep 09 20:11:12.259958 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.259935    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:12.267510 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.267480    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-130.ec2.internal\" not found" node="ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.269741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.269721    2563 manager.go:354] Recovery completed
Sep 09 20:11:12.270250 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.270232    2563 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-130.ec2.internal" not found
Sep 09 20:11:12.274122 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.274109    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:12.275881 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.275866    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:12.275937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.275895    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:12.275937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.275906    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:12.276397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.276379    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:11:12.276397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.276393    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:11:12.276502 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.276410    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:12.276502 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.276395    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:12.277195 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.277178    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:12.277268 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.277206    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:12.277268 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.277217    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:12.277594 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.277580    2563 policy_none.go:49] "None policy: Start"
Sep 09 20:11:12.277594 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.277596    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:11:12.277700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.277607    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:11:12.278609 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.278599    2563 policy_none.go:47] "Start"
Sep 09 20:11:12.287015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.286951    2563 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-130.ec2.internal" not found
Sep 09 20:11:12.320008 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.319845    2563 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.320043    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.320055    2563 server.go:87] "Starting device plugin registration server"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.320287    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.320297    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.320402    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.320488    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.320504    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.321061    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:11:12.321595 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.321113    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.347397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.347382    2563 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-130.ec2.internal" not found
Sep 09 20:11:12.353984 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.353955    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:11:12.355318 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.355297    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:11:12.355387 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.355346    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:11:12.355387 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.355366    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:11:12.355456 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.355400    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:11:12.357721 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.357705    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:12.420886 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.420796    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:12.422408 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.422392    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:12.422472 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.422423    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:12.422472 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.422433    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:12.422472 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.422456    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.431098 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.431085    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.431156 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.431106    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-130.ec2.internal\": node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.446289 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.446260    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.455716 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.455698    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"]
Sep 09 20:11:12.455818 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.455799    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:12.457131 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.457115    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:12.457201 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.457143    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:12.457201 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.457152    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:12.458384 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.458373    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:12.458535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.458517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.458574 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.458553    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:12.459043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.459028    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:12.459106 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.459054    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:12.459106 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.459064    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:12.459182 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.459116    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:12.459182 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.459144    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:12.459182 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.459155    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:12.460246 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.460231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.460285 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.460265    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:12.460901 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.460887    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:12.460980 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.460912    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:12.460980 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.460923    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:12.486405 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.486384    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-130.ec2.internal\" not found" node="ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.490692 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.490677    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-130.ec2.internal\" not found" node="ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.547252 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.547225    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.647505 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.647467    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.654813 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.654777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4404e8f210ca921bf2255cf4cfc0696e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"4404e8f210ca921bf2255cf4cfc0696e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.654910 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.654820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4404e8f210ca921bf2255cf4cfc0696e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"4404e8f210ca921bf2255cf4cfc0696e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.654910 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.654840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcb022e37644c385755d64172316b214-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"fcb022e37644c385755d64172316b214\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.748267 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.748201    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.755541 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.755510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4404e8f210ca921bf2255cf4cfc0696e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"4404e8f210ca921bf2255cf4cfc0696e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.755634 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.755560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4404e8f210ca921bf2255cf4cfc0696e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"4404e8f210ca921bf2255cf4cfc0696e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.755634 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.755594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcb022e37644c385755d64172316b214-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"fcb022e37644c385755d64172316b214\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.755634 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.755620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4404e8f210ca921bf2255cf4cfc0696e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"4404e8f210ca921bf2255cf4cfc0696e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.755742 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.755672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcb022e37644c385755d64172316b214-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"fcb022e37644c385755d64172316b214\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.755742 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.755682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4404e8f210ca921bf2255cf4cfc0696e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"4404e8f210ca921bf2255cf4cfc0696e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.788690 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.788648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.794622 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:12.794602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:12.849034 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.848992    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:12.949574 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:12.949544    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:13.050146 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:13.050066    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:13.150670 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:13.150635    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found"
Sep 09 20:11:13.186500 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.186477    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:11:13.186920 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.186603    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:13.186920 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.186635    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:13.187010 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.186997    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:13.240825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.240797    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:13.247047 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.246999    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:06:13 +0000 UTC" deadline="2028-05-21 16:46:07.384213728 +0000 UTC"
Sep 09 20:11:13.247047 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.247043    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14876h34m54.137173898s"
Sep 09 20:11:13.252735 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.252711    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:13.253195 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.253176    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:13.263671 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.263654    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:13.265126 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.265111    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"
Sep 09 20:11:13.266161 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.266141    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:13.273676 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.273659    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:13.283360 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.283343    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4sxrm"
Sep 09 20:11:13.290145 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.290126    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4sxrm"
Sep 09 20:11:13.328121 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:13.328081    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb022e37644c385755d64172316b214.slice/crio-6c656309939dca7a59dfa798b54c3ceb922e824989a8517a902d268312ac921e WatchSource:0}: Error finding container 6c656309939dca7a59dfa798b54c3ceb922e824989a8517a902d268312ac921e: Status 404 returned error can't find the container with id 6c656309939dca7a59dfa798b54c3ceb922e824989a8517a902d268312ac921e
Sep 09 20:11:13.328690 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:13.328671    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4404e8f210ca921bf2255cf4cfc0696e.slice/crio-67ed5e34ff770ad605c4bfff387eb9ea929bbae47094a3bcedffd07596dcb0cc WatchSource:0}: Error finding container 67ed5e34ff770ad605c4bfff387eb9ea929bbae47094a3bcedffd07596dcb0cc: Status 404 returned error can't find the container with id 67ed5e34ff770ad605c4bfff387eb9ea929bbae47094a3bcedffd07596dcb0cc
Sep 09 20:11:13.333905 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.333885    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:11:13.359220 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.359175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" event={"ID":"4404e8f210ca921bf2255cf4cfc0696e","Type":"ContainerStarted","Data":"67ed5e34ff770ad605c4bfff387eb9ea929bbae47094a3bcedffd07596dcb0cc"}
Sep 09 20:11:13.360205 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:13.360185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" event={"ID":"fcb022e37644c385755d64172316b214","Type":"ContainerStarted","Data":"6c656309939dca7a59dfa798b54c3ceb922e824989a8517a902d268312ac921e"}
Sep 09 20:11:15.297357 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.297270    2563 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:15.306906 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.306878    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:15.307318 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.307291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-m9fx7","kube-system/global-pull-secret-syncer-gz6b2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p","openshift-cluster-node-tuning-operator/tuned-p9zgw","openshift-image-registry/node-ca-rhb6f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal","openshift-multus/multus-additional-cni-plugins-ftsv5","openshift-multus/network-metrics-daemon-jklcc","openshift-network-diagnostics/network-check-target-855rx","kube-system/konnectivity-agent-rcp2l","kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal","openshift-dns/node-resolver-vrpjd","openshift-multus/multus-ntfvh","openshift-network-operator/iptables-alerter-g8c2v"]
Sep 09 20:11:15.308734 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.308710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.309974 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.309951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.311243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.311226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.312259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.311996    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:15.312259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:15.312259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:15.312259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-48d5m\""
Sep 09 20:11:15.312259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:15.312513 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.312513 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312509    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v784q\""
Sep 09 20:11:15.312573 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312546    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:15.312770 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.312509    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:15.314559 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.313972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.314559 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.314330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:15.314559 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.314486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-52qrk\""
Sep 09 20:11:15.315439 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.315413    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:15.315529 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.315447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:15.315592 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.315554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.315910 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.315890    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:15.316862 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.316840    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:15.317440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.317078    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-djfkb\""
Sep 09 20:11:15.317440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.317127    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:15.317440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.317133    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:15.317440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.317130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qrnv2\""
Sep 09 20:11:15.317440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.317322    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:15.319011 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.318991    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:15.319011 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.319008    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:15.319167 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.319143    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:15.319167 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.319150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p2djg\""
Sep 09 20:11:15.319591 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.319571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:15.319696 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.319665    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:15.321142 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.321124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.322392 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.322344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.322462 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.322409    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:15.324068 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.324046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jcwc5\""
Sep 09 20:11:15.324257 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.324238    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:15.324545 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.324529    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:15.324611 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.324594    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:15.324747 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.324732    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:15.324814 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.324794    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:15.324910 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.324531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:15.325114 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.325759 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysctl-d\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.325854 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-hostroot\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.325854 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-multus-certs\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.325854 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-modprobe-d\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.325967 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-systemd\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.325967 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-lib-modules\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.325967 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-var-lib-kubelet\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.325967 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hws58\" (UniqueName: \"kubernetes.io/projected/df2e417e-a206-4060-9bed-17dea319370f-kube-api-access-hws58\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.326093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.325971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.326093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysconfig\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.326093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysctl-conf\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.326093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-sys\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.326093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-tuned\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.326245 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8tx2\" (UniqueName: \"kubernetes.io/projected/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-kube-api-access-t8tx2\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.326245 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df2e417e-a206-4060-9bed-17dea319370f-iptables-alerter-script\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.326245 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dcc119c-416c-460b-bbec-98f4d5d44294-cni-binary-copy\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326245 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-cni-bin\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326245 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-cni-multus\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.326424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:15.326424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-socket-dir-parent\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-registration-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.326424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.326424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.326424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-socket-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6m7\" (UniqueName: \"kubernetes.io/projected/f3f81ce1-51b0-486f-8e23-cb26eefd3636-kube-api-access-7v6m7\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-kubelet\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-cnibin\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-conf-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-daemon-config\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-etc-kubernetes\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-device-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:15.326766 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-cnibin\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-run\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.326791    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-host\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-tmp\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df2e417e-a206-4060-9bed-17dea319370f-host-slash\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-system-cni-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.326976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-netns\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-kubelet-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.327048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-sys-fs\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/163dd371-674d-4aa6-9c0d-11f70d0a5c80-host\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/163dd371-674d-4aa6-9c0d-11f70d0a5c80-serviceca\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td48v\" (UniqueName: \"kubernetes.io/projected/c8455e93-3740-42b5-b358-4d0d0df723a3-kube-api-access-td48v\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-cni-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-k8s-cni-cncf-io\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btsh\" (UniqueName: \"kubernetes.io/projected/5dcc119c-416c-460b-bbec-98f4d5d44294-kube-api-access-7btsh\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.327304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-etc-selinux\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.327737 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88mt\" (UniqueName: \"kubernetes.io/projected/163dd371-674d-4aa6-9c0d-11f70d0a5c80-kube-api-access-l88mt\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.327737 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.327737 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-kubernetes\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.327737 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327581    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:15.327934 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:15.327985 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.327937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-88w5b\""
Sep 09 20:11:15.329722 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.329704    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:15.329914 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.329726    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:15.329914 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.329708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p5b76\""
Sep 09 20:11:15.354498 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.354461    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:14 +0000 UTC" deadline="2028-03-31 11:51:04.216852648 +0000 UTC"
Sep 09 20:11:15.354498 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.354489    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13647h39m48.86236819s"
Sep 09 20:11:15.417389 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.417358    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:15.427898 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.427869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.428068 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.427905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-cni-netd\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.428068 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.427928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e68f0176-5891-423d-ae93-8c6c8cdf1d93-kubelet-config\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.428068 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.427954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-socket-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.428068 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.427995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-cnibin\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-cnibin\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-daemon-config\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-systemd\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-socket-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-cni-bin\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-run\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df2e417e-a206-4060-9bed-17dea319370f-host-slash\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.428243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e68f0176-5891-423d-ae93-8c6c8cdf1d93-dbus\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-sys\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-system-cni-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/163dd371-674d-4aa6-9c0d-11f70d0a5c80-serviceca\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-run\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-slash\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-cni-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df2e417e-a206-4060-9bed-17dea319370f-host-slash\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-etc-selinux\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88mt\" (UniqueName: \"kubernetes.io/projected/163dd371-674d-4aa6-9c0d-11f70d0a5c80-kube-api-access-l88mt\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-systemd\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-lib-modules\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-hosts-file\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.428719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysconfig\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-daemon-config\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-tuned\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8tx2\" (UniqueName: \"kubernetes.io/projected/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-kube-api-access-t8tx2\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dcc119c-416c-460b-bbec-98f4d5d44294-cni-binary-copy\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-cni-multus\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-cni-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysconfig\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-systemd-units\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-node-log\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-sys\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovnkube-script-lib\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-socket-dir-parent\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-lib-modules\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.429346 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-registration-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-etc-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.428989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6m7\" (UniqueName: \"kubernetes.io/projected/f3f81ce1-51b0-486f-8e23-cb26eefd3636-kube-api-access-7v6m7\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-kubelet\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-conf-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-etc-selinux\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-etc-kubernetes\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-tmp\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-kubelet\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovn-node-metrics-cert\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-device-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-cnibin\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-host\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-run-netns\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.430124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8455e93-3740-42b5-b358-4d0d0df723a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429319    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-ovn\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dcc119c-416c-460b-bbec-98f4d5d44294-cni-binary-copy\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-env-overrides\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-netns\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-etc-kubernetes\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-system-cni-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-cnibin\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-kubelet-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-sys-fs\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-host\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/163dd371-674d-4aa6-9c0d-11f70d0a5c80-host\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td48v\" (UniqueName: \"kubernetes.io/projected/c8455e93-3740-42b5-b358-4d0d0df723a3-kube-api-access-td48v\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-conf-dir\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-kubernetes\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-multus-socket-dir-parent\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.430850 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40e16b77-7730-4ae6-95a2-43bae3c36d86-agent-certs\") pod \"konnectivity-agent-rcp2l\" (UID: \"40e16b77-7730-4ae6-95a2-43bae3c36d86\") " pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-kubernetes\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-systemd\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-registration-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-kubelet\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40e16b77-7730-4ae6-95a2-43bae3c36d86-konnectivity-ca\") pod \"konnectivity-agent-rcp2l\" (UID: \"40e16b77-7730-4ae6-95a2-43bae3c36d86\") " pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-device-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-kubelet-dir\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/163dd371-674d-4aa6-9c0d-11f70d0a5c80-host\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3f81ce1-51b0-486f-8e23-cb26eefd3636-sys-fs\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8455e93-3740-42b5-b358-4d0d0df723a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-netns\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-cni-multus\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-k8s-cni-cncf-io\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/163dd371-674d-4aa6-9c0d-11f70d0a5c80-serviceca\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-k8s-cni-cncf-io\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btsh\" (UniqueName: \"kubernetes.io/projected/5dcc119c-416c-460b-bbec-98f4d5d44294-kube-api-access-7btsh\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.431511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysctl-d\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hws58\" (UniqueName: \"kubernetes.io/projected/df2e417e-a206-4060-9bed-17dea319370f-kube-api-access-hws58\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.429994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovnkube-config\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysctl-d\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8khm\" (UniqueName: \"kubernetes.io/projected/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-kube-api-access-b8khm\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f5b5\" (UniqueName: \"kubernetes.io/projected/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-kube-api-access-5f5b5\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-hostroot\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-multus-certs\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-run-multus-certs\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-hostroot\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-modprobe-d\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-var-lib-kubelet\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-tmp-dir\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysctl-conf\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-modprobe-d\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df2e417e-a206-4060-9bed-17dea319370f-iptables-alerter-script\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.432159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-var-lib-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.432819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-var-lib-kubelet\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-sysctl-conf\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.432819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-log-socket\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.432819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5f6\" (UniqueName: \"kubernetes.io/projected/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-kube-api-access-rr5f6\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:15.432819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-cni-bin\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.432819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.430799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5dcc119c-416c-460b-bbec-98f4d5d44294-host-var-lib-cni-bin\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.432819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.431068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df2e417e-a206-4060-9bed-17dea319370f-iptables-alerter-script\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.433109 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.432878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-tmp\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.433372 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.433352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-etc-tuned\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.435629 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.435608    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:15.435769 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.435637    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:15.435769 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.435667    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qj587 for pod openshift-network-diagnostics/network-check-target-855rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:15.435769 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.435716    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587 podName:3dcd3207-d2b6-4acb-8a11-5dffd25b0adf nodeName:}" failed. No retries permitted until 2026-09-09 20:11:15.935701403 +0000 UTC m=+2.974871350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qj587" (UniqueName: "kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587") pod "network-check-target-855rx" (UID: "3dcd3207-d2b6-4acb-8a11-5dffd25b0adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:15.438983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.438962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hws58\" (UniqueName: \"kubernetes.io/projected/df2e417e-a206-4060-9bed-17dea319370f-kube-api-access-hws58\") pod \"iptables-alerter-g8c2v\" (UID: \"df2e417e-a206-4060-9bed-17dea319370f\") " pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.438983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.438978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btsh\" (UniqueName: \"kubernetes.io/projected/5dcc119c-416c-460b-bbec-98f4d5d44294-kube-api-access-7btsh\") pod \"multus-ntfvh\" (UID: \"5dcc119c-416c-460b-bbec-98f4d5d44294\") " pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.439213 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.439194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8tx2\" (UniqueName: \"kubernetes.io/projected/3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e-kube-api-access-t8tx2\") pod \"tuned-p9zgw\" (UID: \"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e\") " pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.439296 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.439196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88mt\" (UniqueName: \"kubernetes.io/projected/163dd371-674d-4aa6-9c0d-11f70d0a5c80-kube-api-access-l88mt\") pod \"node-ca-rhb6f\" (UID: \"163dd371-674d-4aa6-9c0d-11f70d0a5c80\") " pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.440760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.440723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td48v\" (UniqueName: \"kubernetes.io/projected/c8455e93-3740-42b5-b358-4d0d0df723a3-kube-api-access-td48v\") pod \"multus-additional-cni-plugins-ftsv5\" (UID: \"c8455e93-3740-42b5-b358-4d0d0df723a3\") " pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.441372 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.441354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6m7\" (UniqueName: \"kubernetes.io/projected/f3f81ce1-51b0-486f-8e23-cb26eefd3636-kube-api-access-7v6m7\") pod \"aws-ebs-csi-driver-node-js55p\" (UID: \"f3f81ce1-51b0-486f-8e23-cb26eefd3636\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.531840 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.531811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-var-lib-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.531840 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.531842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-log-socket\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532021 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.531858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5f6\" (UniqueName: \"kubernetes.io/projected/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-kube-api-access-rr5f6\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:15.532021 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.531930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-log-socket\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532021 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.531931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-var-lib-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-cni-netd\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e68f0176-5891-423d-ae93-8c6c8cdf1d93-kubelet-config\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.532112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-systemd\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-cni-netd\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-cni-bin\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-cni-bin\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.532206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e68f0176-5891-423d-ae93-8c6c8cdf1d93-kubelet-config\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.532206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-systemd\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e68f0176-5891-423d-ae93-8c6c8cdf1d93-dbus\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.532260    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e68f0176-5891-423d-ae93-8c6c8cdf1d93-dbus\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-slash\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.532323    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret podName:e68f0176-5891-423d-ae93-8c6c8cdf1d93 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.032303419 +0000 UTC m=+3.071473362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret") pod "global-pull-secret-syncer-gz6b2" (UID: "e68f0176-5891-423d-ae93-8c6c8cdf1d93") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-slash\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-hosts-file\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-systemd-units\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-node-log\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-systemd-units\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-node-log\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovnkube-script-lib\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-etc-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-hosts-file\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-etc-openvswitch\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-kubelet\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovn-node-metrics-cert\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-kubelet\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-run-netns\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-host-run-netns\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-ovn\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.532903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-env-overrides\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-run-ovn\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40e16b77-7730-4ae6-95a2-43bae3c36d86-agent-certs\") pod \"konnectivity-agent-rcp2l\" (UID: \"40e16b77-7730-4ae6-95a2-43bae3c36d86\") " pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40e16b77-7730-4ae6-95a2-43bae3c36d86-konnectivity-ca\") pod \"konnectivity-agent-rcp2l\" (UID: \"40e16b77-7730-4ae6-95a2-43bae3c36d86\") " pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovnkube-config\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8khm\" (UniqueName: \"kubernetes.io/projected/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-kube-api-access-b8khm\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f5b5\" (UniqueName: \"kubernetes.io/projected/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-kube-api-access-5f5b5\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.532933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-tmp-dir\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.533221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-tmp-dir\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.533305    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.533351    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs podName:a5c6873b-0047-43fc-a158-8ff4ceb8e0dd nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.033335458 +0000 UTC m=+3.072505388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs") pod "network-metrics-daemon-jklcc" (UID: "a5c6873b-0047-43fc-a158-8ff4ceb8e0dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.533673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40e16b77-7730-4ae6-95a2-43bae3c36d86-konnectivity-ca\") pod \"konnectivity-agent-rcp2l\" (UID: \"40e16b77-7730-4ae6-95a2-43bae3c36d86\") " pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.533782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovnkube-script-lib\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.533893 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.533817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-env-overrides\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.534536 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.533939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovnkube-config\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.535203 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.535181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-ovn-node-metrics-cert\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.535363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.535344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40e16b77-7730-4ae6-95a2-43bae3c36d86-agent-certs\") pod \"konnectivity-agent-rcp2l\" (UID: \"40e16b77-7730-4ae6-95a2-43bae3c36d86\") " pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.540618 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.540589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5f6\" (UniqueName: \"kubernetes.io/projected/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-kube-api-access-rr5f6\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:15.541267 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.541247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f5b5\" (UniqueName: \"kubernetes.io/projected/d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef-kube-api-access-5f5b5\") pod \"node-resolver-vrpjd\" (UID: \"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef\") " pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.541349 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.541301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8khm\" (UniqueName: \"kubernetes.io/projected/e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da-kube-api-access-b8khm\") pod \"ovnkube-node-m9fx7\" (UID: \"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.622515 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.622427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ftsv5"
Sep 09 20:11:15.630551 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.630528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ntfvh"
Sep 09 20:11:15.640491 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.640467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p"
Sep 09 20:11:15.644187 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.644166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p9zgw"
Sep 09 20:11:15.651741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.651716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rhb6f"
Sep 09 20:11:15.658338 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.658322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g8c2v"
Sep 09 20:11:15.664959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.664940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:15.672585 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.672561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:15.677590 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.677571    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:15.679715 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.679696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vrpjd"
Sep 09 20:11:15.816832 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.816793    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:15.935871 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:15.935797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:15.936004 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.935980    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:15.936004 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.936002    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:15.936073 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.936012    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qj587 for pod openshift-network-diagnostics/network-check-target-855rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:15.936073 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:15.936063    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587 podName:3dcd3207-d2b6-4acb-8a11-5dffd25b0adf nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.936050981 +0000 UTC m=+3.975220910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj587" (UniqueName: "kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587") pod "network-check-target-855rx" (UID: "3dcd3207-d2b6-4acb-8a11-5dffd25b0adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:16.036752 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.036711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:16.036909 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.036762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:16.036909 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.036869    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:16.036909 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.036884    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:16.037022 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.036921    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret podName:e68f0176-5891-423d-ae93-8c6c8cdf1d93 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:17.036908095 +0000 UTC m=+4.076078024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret") pod "global-pull-secret-syncer-gz6b2" (UID: "e68f0176-5891-423d-ae93-8c6c8cdf1d93") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:16.037022 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.036951    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs podName:a5c6873b-0047-43fc-a158-8ff4ceb8e0dd nodeName:}" failed. No retries permitted until 2026-09-09 20:11:17.036932853 +0000 UTC m=+4.076102801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs") pod "network-metrics-daemon-jklcc" (UID: "a5c6873b-0047-43fc-a158-8ff4ceb8e0dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:16.182980 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:16.182950    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f81ce1_51b0_486f_8e23_cb26eefd3636.slice/crio-fd8711d077ac86962bb17bf3dc8d5532cb9c71dc14bd4e8bb6031cb9ee4595b9 WatchSource:0}: Error finding container fd8711d077ac86962bb17bf3dc8d5532cb9c71dc14bd4e8bb6031cb9ee4595b9: Status 404 returned error can't find the container with id fd8711d077ac86962bb17bf3dc8d5532cb9c71dc14bd4e8bb6031cb9ee4595b9
Sep 09 20:11:16.184311 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:16.184284    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e7c5d4_bd8e_4d59_94d9_cd43edc64cef.slice/crio-455b7d737916f0d44f53b11589b1d26e2ab55dfe35990c8e66ab87e179de0419 WatchSource:0}: Error finding container 455b7d737916f0d44f53b11589b1d26e2ab55dfe35990c8e66ab87e179de0419: Status 404 returned error can't find the container with id 455b7d737916f0d44f53b11589b1d26e2ab55dfe35990c8e66ab87e179de0419
Sep 09 20:11:16.184855 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:16.184831    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba1d7bf_58a0_4113_82bc_e0a5183e3f5e.slice/crio-63145053fb756babc492019f2d38e18f88a56294d18d19ad83534079e01a9baf WatchSource:0}: Error finding container 63145053fb756babc492019f2d38e18f88a56294d18d19ad83534079e01a9baf: Status 404 returned error can't find the container with id 63145053fb756babc492019f2d38e18f88a56294d18d19ad83534079e01a9baf
Sep 09 20:11:16.185777 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:16.185753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e16b77_7730_4ae6_95a2_43bae3c36d86.slice/crio-cb33a2ee72bd6c54ce664a4b56fb948981482a09c58c666c15fd5266ff3f8056 WatchSource:0}: Error finding container cb33a2ee72bd6c54ce664a4b56fb948981482a09c58c666c15fd5266ff3f8056: Status 404 returned error can't find the container with id cb33a2ee72bd6c54ce664a4b56fb948981482a09c58c666c15fd5266ff3f8056
Sep 09 20:11:16.188082 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:16.188063    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcc119c_416c_460b_bbec_98f4d5d44294.slice/crio-b661be87a708c212e6f1385e8fd55dd537b17d90333feafcbd8b7955ad1b23c0 WatchSource:0}: Error finding container b661be87a708c212e6f1385e8fd55dd537b17d90333feafcbd8b7955ad1b23c0: Status 404 returned error can't find the container with id b661be87a708c212e6f1385e8fd55dd537b17d90333feafcbd8b7955ad1b23c0
Sep 09 20:11:16.189456 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:16.189317    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8455e93_3740_42b5_b358_4d0d0df723a3.slice/crio-37224657ebdd4a449db20bab1cfc74ce60cd512be378e2fbf7081588385f7d65 WatchSource:0}: Error finding container 37224657ebdd4a449db20bab1cfc74ce60cd512be378e2fbf7081588385f7d65: Status 404 returned error can't find the container with id 37224657ebdd4a449db20bab1cfc74ce60cd512be378e2fbf7081588385f7d65
Sep 09 20:11:16.196907 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:11:16.196876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163dd371_674d_4aa6_9c0d_11f70d0a5c80.slice/crio-da5cef2b947dc518d42a259735755c1be1e9a2f0799bba2a64622ffe0b8d429a WatchSource:0}: Error finding container da5cef2b947dc518d42a259735755c1be1e9a2f0799bba2a64622ffe0b8d429a: Status 404 returned error can't find the container with id da5cef2b947dc518d42a259735755c1be1e9a2f0799bba2a64622ffe0b8d429a
Sep 09 20:11:16.355083 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.355032    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:14 +0000 UTC" deadline="2028-05-06 10:44:17.737174119 +0000 UTC"
Sep 09 20:11:16.355083 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.355078    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14510h33m1.38209909s"
Sep 09 20:11:16.419044 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.418879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:16.419172 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.419151    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:16.427869 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.427839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" event={"ID":"fcb022e37644c385755d64172316b214","Type":"ContainerStarted","Data":"037d82968386897f61cc0f9b608416fc319ddb0eb7f9d3eccd59d895efa88260"}
Sep 09 20:11:16.428967 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.428946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rhb6f" event={"ID":"163dd371-674d-4aa6-9c0d-11f70d0a5c80","Type":"ContainerStarted","Data":"da5cef2b947dc518d42a259735755c1be1e9a2f0799bba2a64622ffe0b8d429a"}
Sep 09 20:11:16.430107 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.430088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"c354c809a59a4af2ac0c248ee4ffc3648276c64f1ce19f5cb920bb419c8ccb3a"}
Sep 09 20:11:16.431870 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.431847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ntfvh" event={"ID":"5dcc119c-416c-460b-bbec-98f4d5d44294","Type":"ContainerStarted","Data":"b661be87a708c212e6f1385e8fd55dd537b17d90333feafcbd8b7955ad1b23c0"}
Sep 09 20:11:16.433058 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.433035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcp2l" event={"ID":"40e16b77-7730-4ae6-95a2-43bae3c36d86","Type":"ContainerStarted","Data":"cb33a2ee72bd6c54ce664a4b56fb948981482a09c58c666c15fd5266ff3f8056"}
Sep 09 20:11:16.434149 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.434121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g8c2v" event={"ID":"df2e417e-a206-4060-9bed-17dea319370f","Type":"ContainerStarted","Data":"80558a071c95449b52f715e05cdd416121f929f407fbde8e9dbf27f7b8b905f0"}
Sep 09 20:11:16.435297 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.435276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerStarted","Data":"37224657ebdd4a449db20bab1cfc74ce60cd512be378e2fbf7081588385f7d65"}
Sep 09 20:11:16.436420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.436371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p9zgw" event={"ID":"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e","Type":"ContainerStarted","Data":"63145053fb756babc492019f2d38e18f88a56294d18d19ad83534079e01a9baf"}
Sep 09 20:11:16.437453 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.437434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vrpjd" event={"ID":"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef","Type":"ContainerStarted","Data":"455b7d737916f0d44f53b11589b1d26e2ab55dfe35990c8e66ab87e179de0419"}
Sep 09 20:11:16.438510 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.438492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p" event={"ID":"f3f81ce1-51b0-486f-8e23-cb26eefd3636","Type":"ContainerStarted","Data":"fd8711d077ac86962bb17bf3dc8d5532cb9c71dc14bd4e8bb6031cb9ee4595b9"}
Sep 09 20:11:16.439956 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.439926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" event={"ID":"4404e8f210ca921bf2255cf4cfc0696e","Type":"ContainerStarted","Data":"f1994f801c1436fd46ddcbf89cf6676d797cda4823b6f6719a2c7fe839c53aed"}
Sep 09 20:11:16.453148 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.453099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" podStartSLOduration=2.453085399 podStartE2EDuration="2.453085399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:16.440084848 +0000 UTC m=+3.479254802" watchObservedRunningTime="2026-09-09 20:11:16.453085399 +0000 UTC m=+3.492255353"
Sep 09 20:11:16.943859 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:16.943815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:16.944050 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.943995    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:16.944050 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.944016    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:16.944050 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.944030    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qj587 for pod openshift-network-diagnostics/network-check-target-855rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:16.944252 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:16.944089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587 podName:3dcd3207-d2b6-4acb-8a11-5dffd25b0adf nodeName:}" failed. No retries permitted until 2026-09-09 20:11:18.944070831 +0000 UTC m=+5.983240766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj587" (UniqueName: "kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587") pod "network-check-target-855rx" (UID: "3dcd3207-d2b6-4acb-8a11-5dffd25b0adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:17.044755 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:17.044715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:17.044923 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:17.044775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:17.044923 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:17.044898    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:17.045049 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:17.044965    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret podName:e68f0176-5891-423d-ae93-8c6c8cdf1d93 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:19.044946376 +0000 UTC m=+6.084116310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret") pod "global-pull-secret-syncer-gz6b2" (UID: "e68f0176-5891-423d-ae93-8c6c8cdf1d93") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:17.045049 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:17.045032    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:17.045158 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:17.045072    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs podName:a5c6873b-0047-43fc-a158-8ff4ceb8e0dd nodeName:}" failed. No retries permitted until 2026-09-09 20:11:19.045058684 +0000 UTC m=+6.084228616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs") pod "network-metrics-daemon-jklcc" (UID: "a5c6873b-0047-43fc-a158-8ff4ceb8e0dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:17.419074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:17.418535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:17.419074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:17.418558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:17.419074 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:17.418691    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:17.419074 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:17.418822    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:17.461571 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:17.461482    2563 generic.go:356] "Generic (PLEG): container finished" podID="4404e8f210ca921bf2255cf4cfc0696e" containerID="f1994f801c1436fd46ddcbf89cf6676d797cda4823b6f6719a2c7fe839c53aed" exitCode=0
Sep 09 20:11:17.462519 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:17.462490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" event={"ID":"4404e8f210ca921bf2255cf4cfc0696e","Type":"ContainerDied","Data":"f1994f801c1436fd46ddcbf89cf6676d797cda4823b6f6719a2c7fe839c53aed"}
Sep 09 20:11:18.419206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:18.419164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:18.419749 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:18.419309    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:18.486118 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:18.486083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" event={"ID":"4404e8f210ca921bf2255cf4cfc0696e","Type":"ContainerStarted","Data":"8657dd21ca6cbc1d6acb5f0718e6e6ed45bcccac6a309b755b1af5cd99fd10f5"}
Sep 09 20:11:18.958866 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:18.958820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:18.959041 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:18.959029    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:18.959117 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:18.959050    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:18.959117 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:18.959063    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qj587 for pod openshift-network-diagnostics/network-check-target-855rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:18.959234 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:18.959123    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587 podName:3dcd3207-d2b6-4acb-8a11-5dffd25b0adf nodeName:}" failed. No retries permitted until 2026-09-09 20:11:22.959105345 +0000 UTC m=+9.998275280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj587" (UniqueName: "kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587") pod "network-check-target-855rx" (UID: "3dcd3207-d2b6-4acb-8a11-5dffd25b0adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:19.059970 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:19.059905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:19.060148 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:19.060026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:19.060225 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:19.060171    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:19.060280 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:19.060237    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs podName:a5c6873b-0047-43fc-a158-8ff4ceb8e0dd nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.060217467 +0000 UTC m=+10.099387419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs") pod "network-metrics-daemon-jklcc" (UID: "a5c6873b-0047-43fc-a158-8ff4ceb8e0dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:19.060669 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:19.060633    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:19.060763 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:19.060705    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret podName:e68f0176-5891-423d-ae93-8c6c8cdf1d93 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.060688464 +0000 UTC m=+10.099858398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret") pod "global-pull-secret-syncer-gz6b2" (UID: "e68f0176-5891-423d-ae93-8c6c8cdf1d93") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:19.418624 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:19.418539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:19.418819 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:19.418688    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:19.421113 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:19.421091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:19.421545 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:19.421219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:20.418672 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:20.418622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:20.418868 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:20.418825    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:21.419386 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:21.419351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:21.419943 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:21.419499    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:21.419943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:21.419880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:21.420074 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:21.420047    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:22.419458 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:22.419358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:22.419951 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:22.419491    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:22.994146 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:22.994106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:22.994351 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:22.994332    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:22.994426 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:22.994360    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:22.994426 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:22.994373    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qj587 for pod openshift-network-diagnostics/network-check-target-855rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:22.994540 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:22.994430    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587 podName:3dcd3207-d2b6-4acb-8a11-5dffd25b0adf nodeName:}" failed. No retries permitted until 2026-09-09 20:11:30.994411415 +0000 UTC m=+18.033581351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj587" (UniqueName: "kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587") pod "network-check-target-855rx" (UID: "3dcd3207-d2b6-4acb-8a11-5dffd25b0adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:23.095284 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:23.095250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:23.095452 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:23.095296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:23.095452 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:23.095416    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:23.095609 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:23.095476    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret podName:e68f0176-5891-423d-ae93-8c6c8cdf1d93 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:31.095457655 +0000 UTC m=+18.134627589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret") pod "global-pull-secret-syncer-gz6b2" (UID: "e68f0176-5891-423d-ae93-8c6c8cdf1d93") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:23.095609 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:23.095503    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:23.095609 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:23.095567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs podName:a5c6873b-0047-43fc-a158-8ff4ceb8e0dd nodeName:}" failed. No retries permitted until 2026-09-09 20:11:31.095548984 +0000 UTC m=+18.134718917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs") pod "network-metrics-daemon-jklcc" (UID: "a5c6873b-0047-43fc-a158-8ff4ceb8e0dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:23.421911 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:23.421833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:23.422241 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:23.421958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:23.422241 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:23.421963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:23.422241 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:23.422046    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:24.418965 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:24.418455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:24.418965 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:24.418596    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:25.420136 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:25.419582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:25.420136 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:25.419581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:25.420136 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:25.419754    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:25.420136 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:25.419790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:26.419249 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:26.419209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:26.419419 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:26.419339    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:27.419013 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:27.418978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:27.419447 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:27.418983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:27.419447 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:27.419139    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:27.419447 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:27.419227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:28.418464 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:28.418423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:28.418635 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:28.418541    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:29.419059 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:29.419020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:29.419439 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:29.419149    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:29.419439 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:29.419205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:29.419439 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:29.419324    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:30.418712 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:30.418672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:30.418969 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:30.418789    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:31.051596 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:31.051508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:31.052028 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.051730    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:31.052028 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.051760    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:31.052028 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.051776    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qj587 for pod openshift-network-diagnostics/network-check-target-855rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:31.052028 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.051842    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587 podName:3dcd3207-d2b6-4acb-8a11-5dffd25b0adf nodeName:}" failed. No retries permitted until 2026-09-09 20:11:47.05182363 +0000 UTC m=+34.090993562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj587" (UniqueName: "kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587") pod "network-check-target-855rx" (UID: "3dcd3207-d2b6-4acb-8a11-5dffd25b0adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:31.152095 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:31.152057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:31.152289 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:31.152152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:31.152289 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.152225    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:31.152289 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.152251    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:31.152427 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.152305    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret podName:e68f0176-5891-423d-ae93-8c6c8cdf1d93 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:47.152284082 +0000 UTC m=+34.191454030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret") pod "global-pull-secret-syncer-gz6b2" (UID: "e68f0176-5891-423d-ae93-8c6c8cdf1d93") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:31.152427 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.152323    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs podName:a5c6873b-0047-43fc-a158-8ff4ceb8e0dd nodeName:}" failed. No retries permitted until 2026-09-09 20:11:47.152315415 +0000 UTC m=+34.191485343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs") pod "network-metrics-daemon-jklcc" (UID: "a5c6873b-0047-43fc-a158-8ff4ceb8e0dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:31.419262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:31.419189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:31.419262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:31.419219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:31.419479 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.419313    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:31.419479 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:31.419418    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:32.419158 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:32.419116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:32.419574 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:32.419275    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:33.419469 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:33.419440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:33.419886 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:33.419536    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:33.419886 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:33.419589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:33.419886 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:33.419714    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:34.418849 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:34.418811    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:34.419023 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:34.419003    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:34.526085 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:34.526046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rhb6f" event={"ID":"163dd371-674d-4aa6-9c0d-11f70d0a5c80","Type":"ContainerStarted","Data":"cc63c6610316a6ad01aa6f7f351c6bd987d4f4454ed09c025b9f1722d4a0d086"}
Sep 09 20:11:34.530772 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:34.530343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcp2l" event={"ID":"40e16b77-7730-4ae6-95a2-43bae3c36d86","Type":"ContainerStarted","Data":"661731eb24444f6ff669f582671c1d1a25cb1219ee1c09a769d854b8d7ee70ad"}
Sep 09 20:11:34.544312 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:34.544258    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rhb6f" podStartSLOduration=3.480326446 podStartE2EDuration="21.544241029s" totalImagesPullingTime="18.063914583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:34.543851204 +0000 UTC m=+21.583021156" watchObservedRunningTime="2026-09-09 20:11:34.544241029 +0000 UTC m=+21.583410980"
Sep 09 20:11:34.544423 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:34.544363    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" podStartSLOduration=20.544355331 podStartE2EDuration="20.544355331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:18.502596786 +0000 UTC m=+5.541766739" watchObservedRunningTime="2026-09-09 20:11:34.544355331 +0000 UTC m=+21.583525282"
Sep 09 20:11:34.561278 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:34.561229    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rcp2l" podStartSLOduration=6.969872095 podStartE2EDuration="21.561208186s" totalImagesPullingTime="14.591336091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:34.560601905 +0000 UTC m=+21.599771886" watchObservedRunningTime="2026-09-09 20:11:34.561208186 +0000 UTC m=+21.600378138"
Sep 09 20:11:35.350922 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.350600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:35.351328 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.351311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:35.419279 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.419244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:35.419436 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:35.419364    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:35.419436 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.419412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:35.419547 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:35.419511    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:35.534068 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.534035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vrpjd" event={"ID":"d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef","Type":"ContainerStarted","Data":"2432992efebdfb55544c7183c11250a9ba13b907fd6ca59480d07f58896e19cc"}
Sep 09 20:11:35.535556 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.535530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p" event={"ID":"f3f81ce1-51b0-486f-8e23-cb26eefd3636","Type":"ContainerStarted","Data":"e16123301a599c5b7f294659bd0943899b94a961b5870d94e621d0f566161ece"}
Sep 09 20:11:35.538580 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.538558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"e2cd25bc86e47ec31fcb6147291d4cbcff3ff9a5882aeecbd11e8f08fd618c8e"}
Sep 09 20:11:35.538702 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.538613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"c7623bf08166052af72803ae1974f1e636ff39684c4ef8995d2d63658e92008a"}
Sep 09 20:11:35.538702 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.538676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"c0b10fba92c07bb4855237f1183e30afa68bdc68c1d90fca395c6924d7305ef0"}
Sep 09 20:11:35.538812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.538714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"e37c1b99206fe636551b0b1d0050ea874f528e20619f41284a24390ecb5e46af"}
Sep 09 20:11:35.538812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.538747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"e2547988b7b2ffd0c24d0ea1b356db38be4b7ee11360b373d102ac261eb0578a"}
Sep 09 20:11:35.538812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.538783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"4c31be82ef7780d36e7e8e050e8a79e9d6ff1eb4995cc0efec17668da346f550"}
Sep 09 20:11:35.539926 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.539903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ntfvh" event={"ID":"5dcc119c-416c-460b-bbec-98f4d5d44294","Type":"ContainerStarted","Data":"09543cdc5ecf8312c6128250d6704dab3a31c6fcdafa8a62ff28c5e31cfd7893"}
Sep 09 20:11:35.541256 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.541235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g8c2v" event={"ID":"df2e417e-a206-4060-9bed-17dea319370f","Type":"ContainerStarted","Data":"8b09dc0cd43061ba7852a0a59d41068ed404cbca162fa22da6322cfe4ac52134"}
Sep 09 20:11:35.542730 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.542708    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8455e93-3740-42b5-b358-4d0d0df723a3" containerID="b877129070f3e001c12ad94f0529204a9b40b5cbd43812bfd3ebb193a75c6669" exitCode=0
Sep 09 20:11:35.542817 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.542790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerDied","Data":"b877129070f3e001c12ad94f0529204a9b40b5cbd43812bfd3ebb193a75c6669"}
Sep 09 20:11:35.544306 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.544258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p9zgw" event={"ID":"3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e","Type":"ContainerStarted","Data":"7d329de3270b7b326b5b818cc789d81e652355f51656f20245d284146f6a4736"}
Sep 09 20:11:35.544559 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.544536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:35.544904 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.544887    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rcp2l"
Sep 09 20:11:35.548080 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.548047    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vrpjd" podStartSLOduration=4.466256626 podStartE2EDuration="22.548036092s" totalImagesPullingTime="18.081779466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:35.547772436 +0000 UTC m=+22.586942387" watchObservedRunningTime="2026-09-09 20:11:35.548036092 +0000 UTC m=+22.587206040"
Sep 09 20:11:35.564046 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.564003    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ntfvh" podStartSLOduration=4.494686454 podStartE2EDuration="22.563991352s" totalImagesPullingTime="18.069304898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:35.563391746 +0000 UTC m=+22.602561696" watchObservedRunningTime="2026-09-09 20:11:35.563991352 +0000 UTC m=+22.603161302"
Sep 09 20:11:35.578219 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.578180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p9zgw" podStartSLOduration=4.501005483 podStartE2EDuration="22.578167588s" totalImagesPullingTime="18.077162105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:35.57801502 +0000 UTC m=+22.617184970" watchObservedRunningTime="2026-09-09 20:11:35.578167588 +0000 UTC m=+22.617337538"
Sep 09 20:11:35.627367 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:35.627275    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g8c2v" podStartSLOduration=4.643822824 podStartE2EDuration="22.627261068s" totalImagesPullingTime="17.983438244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:35.613748446 +0000 UTC m=+22.652918397" watchObservedRunningTime="2026-09-09 20:11:35.627261068 +0000 UTC m=+22.666431000"
Sep 09 20:11:36.250212 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:36.249836    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:36.397966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:36.397815    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:36.24985975Z","UUID":"8a28eafb-2141-4bd6-805b-271580244713","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:36.399405 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:36.399371    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:36.399405 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:36.399407    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:36.419052 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:36.418715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:36.419052 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:36.418818    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:36.549573 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:36.549533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p" event={"ID":"f3f81ce1-51b0-486f-8e23-cb26eefd3636","Type":"ContainerStarted","Data":"38f7ec236330541fc73da05deca98470f50d421dcb44cd4dc9a5c9fe84f33404"}
Sep 09 20:11:37.418805 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:37.418764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:37.418985 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:37.418929    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:37.419412 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:37.419383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:37.419554 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:37.419501    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:37.555901 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:37.555863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"b6e9bda5d024bfa1bb67fdb7cfbe5c84d3e1413440ccafa0301527314ec95a28"}
Sep 09 20:11:38.418785 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:38.418752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:38.418964 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:38.418890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:38.560077 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:38.560039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p" event={"ID":"f3f81ce1-51b0-486f-8e23-cb26eefd3636","Type":"ContainerStarted","Data":"a684bb8718445bf41188b7f015dc4dd222d0218d3593d8b9dbd4616cb5567f48"}
Sep 09 20:11:38.578265 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:38.578219    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-js55p" podStartSLOduration=4.729165026 podStartE2EDuration="25.57820389s" totalImagesPullingTime="20.849038864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:38.576571808 +0000 UTC m=+25.615741760" watchObservedRunningTime="2026-09-09 20:11:38.57820389 +0000 UTC m=+25.617373841"
Sep 09 20:11:39.421035 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:39.419368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:39.421035 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:39.419370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:39.421035 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:39.419764    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:39.421035 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:39.419812    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:40.418485 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.418444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:40.419471 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:40.418572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:40.566053 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.566008    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8455e93-3740-42b5-b358-4d0d0df723a3" containerID="fdb7b59e1c6bdc4fcbdca180f3f1c49c884fadf22c450b0907d726f1ce03a351" exitCode=0
Sep 09 20:11:40.566237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.566077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerDied","Data":"fdb7b59e1c6bdc4fcbdca180f3f1c49c884fadf22c450b0907d726f1ce03a351"}
Sep 09 20:11:40.570151 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.570123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" event={"ID":"e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da","Type":"ContainerStarted","Data":"c981e88f42e5733287f544b3edc394b4f31b11dcea535879befc2b3a62c951e8"}
Sep 09 20:11:40.570467 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.570450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:40.587414 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.587385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:40.620097 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.620056    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7" podStartSLOduration=9.263627097 podStartE2EDuration="27.620041839s" totalImagesPullingTime="18.356414742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:40.619806031 +0000 UTC m=+27.658975981" watchObservedRunningTime="2026-09-09 20:11:40.620041839 +0000 UTC m=+27.659211789"
Sep 09 20:11:40.817998 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:40.817973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vrpjd_d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef/dns-node-resolver/0.log"
Sep 09 20:11:41.419507 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.419315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:41.420078 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.419315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:41.420078 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:41.419625    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:41.420078 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:41.419714    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:41.573210 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.573182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:41.573495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.573260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:41.588678 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.588635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:11:41.722546 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.722447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-855rx"]
Sep 09 20:11:41.722733 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.722583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:41.722790 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:41.722724    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:41.723186 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.723164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gz6b2"]
Sep 09 20:11:41.723310 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.723248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:41.723365 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:41.723346    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:41.723853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.723834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jklcc"]
Sep 09 20:11:41.723956 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.723920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:41.724015 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:41.724000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:41.992178 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:41.992099    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rhb6f_163dd371-674d-4aa6-9c0d-11f70d0a5c80/node-ca/0.log"
Sep 09 20:11:42.577402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:42.577357    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8455e93-3740-42b5-b358-4d0d0df723a3" containerID="ddb11664eca6600c753fb815e754c4d627de8a772227ec8e4eaabae80ff58296" exitCode=0
Sep 09 20:11:42.577819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:42.577444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerDied","Data":"ddb11664eca6600c753fb815e754c4d627de8a772227ec8e4eaabae80ff58296"}
Sep 09 20:11:43.419622 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:43.419598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:43.419773 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:43.419695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:43.419773 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:43.419743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:43.419877 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:43.419834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:43.420009 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:43.419980    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:43.420111 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:43.420011    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:43.585518 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:43.584056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerStarted","Data":"99633275ab63cb989c9e4f968dd5b20861187321956a5af52324358ae16a43f9"}
Sep 09 20:11:44.591866 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:44.591640    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8455e93-3740-42b5-b358-4d0d0df723a3" containerID="99633275ab63cb989c9e4f968dd5b20861187321956a5af52324358ae16a43f9" exitCode=0
Sep 09 20:11:44.591866 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:44.591687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerDied","Data":"99633275ab63cb989c9e4f968dd5b20861187321956a5af52324358ae16a43f9"}
Sep 09 20:11:45.421349 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:45.421321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:45.421517 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:45.421439    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:45.421788 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:45.421763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:45.421911 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:45.421785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:45.421911 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:45.421857    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:45.421990 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:45.421932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:47.072385 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:47.072343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:47.072923 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.072493    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:47.072923 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.072517    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:47.072923 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.072527    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qj587 for pod openshift-network-diagnostics/network-check-target-855rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:47.072923 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.072588    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587 podName:3dcd3207-d2b6-4acb-8a11-5dffd25b0adf nodeName:}" failed. No retries permitted until 2026-09-09 20:12:19.072570036 +0000 UTC m=+66.111739983 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj587" (UniqueName: "kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587") pod "network-check-target-855rx" (UID: "3dcd3207-d2b6-4acb-8a11-5dffd25b0adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:47.173279 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:47.173236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:47.173463 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:47.173300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:47.173463 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.173419    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:47.173578 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.173462    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:47.173578 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.173498    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs podName:a5c6873b-0047-43fc-a158-8ff4ceb8e0dd nodeName:}" failed. No retries permitted until 2026-09-09 20:12:19.173478835 +0000 UTC m=+66.212648786 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs") pod "network-metrics-daemon-jklcc" (UID: "a5c6873b-0047-43fc-a158-8ff4ceb8e0dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:47.173578 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.173520    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret podName:e68f0176-5891-423d-ae93-8c6c8cdf1d93 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:19.173505936 +0000 UTC m=+66.212675868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret") pod "global-pull-secret-syncer-gz6b2" (UID: "e68f0176-5891-423d-ae93-8c6c8cdf1d93") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:47.418797 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:47.418715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:47.418797 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:47.418738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:47.419002 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:47.418727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:47.419002 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.418859    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:47.419002 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.418986    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:47.419124 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:47.419093    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:49.418812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:49.418773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:49.418812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:49.418811    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:49.419392 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:49.418771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:49.419392 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:49.418890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:49.419392 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:49.418992    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:49.419392 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:49.419054    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:49.607257 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:49.607225    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8455e93-3740-42b5-b358-4d0d0df723a3" containerID="385e6377f57d1d21d05190c9de49c89a67b7cd4c13d2cfcad1e37bbc45390164" exitCode=0
Sep 09 20:11:49.607257 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:49.607262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerDied","Data":"385e6377f57d1d21d05190c9de49c89a67b7cd4c13d2cfcad1e37bbc45390164"}
Sep 09 20:11:50.612990 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:50.612960    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8455e93-3740-42b5-b358-4d0d0df723a3" containerID="70c14c602a04597200fdbde77727b9213b5881b9dee0b18877ff861a9deb8c19" exitCode=0
Sep 09 20:11:50.613433 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:50.613008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerDied","Data":"70c14c602a04597200fdbde77727b9213b5881b9dee0b18877ff861a9deb8c19"}
Sep 09 20:11:51.419017 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:51.418982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:51.419191 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:51.418990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:51.419191 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:51.419085    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:51.419191 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:51.418992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:51.419359 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:51.419155    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:51.419359 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:51.419272    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:51.619160 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:51.619127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" event={"ID":"c8455e93-3740-42b5-b358-4d0d0df723a3","Type":"ContainerStarted","Data":"6a67eba5b83a973ccb959cf28074e056e7ca0f46b91c04fc85adde606075251e"}
Sep 09 20:11:51.642427 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:51.642378    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ftsv5" podStartSLOduration=10.131373221 podStartE2EDuration="38.64236297s" totalImagesPullingTime="28.510989749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:51.64092189 +0000 UTC m=+38.680091867" watchObservedRunningTime="2026-09-09 20:11:51.64236297 +0000 UTC m=+38.681532918"
Sep 09 20:11:53.419967 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:53.419933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:53.420560 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:53.420006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:53.420560 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:53.420017    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:53.420560 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:53.420138    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:53.420560 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:53.420167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:53.420560 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:53.420233    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:55.418947 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:55.418913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:55.419404 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:55.418913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:55.419404 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:55.419026    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:55.419404 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:55.418921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:55.419404 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:55.419105    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:55.419404 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:55.419208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:57.419216 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:57.419179    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:57.419587 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:57.419181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:57.419587 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:57.419296    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:57.419587 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:57.419369    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:57.419587 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:57.419179    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:57.419587 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:57.419451    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:11:59.418623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:59.418590    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:11:59.418623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:59.418607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:11:59.418623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:11:59.418590    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:11:59.419240 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:59.418723    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:11:59.419240 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:59.418772    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:11:59.419240 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:11:59.418846    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:12:01.418839 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:01.418796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:12:01.419326 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:01.418796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:12:01.419326 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:12:01.418922    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jklcc" podUID="a5c6873b-0047-43fc-a158-8ff4ceb8e0dd"
Sep 09 20:12:01.419326 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:01.418795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:12:01.419326 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:12:01.418979    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-855rx" podUID="3dcd3207-d2b6-4acb-8a11-5dffd25b0adf"
Sep 09 20:12:01.419326 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:12:01.419049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz6b2" podUID="e68f0176-5891-423d-ae93-8c6c8cdf1d93"
Sep 09 20:12:03.249852 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.249783    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeReady"
Sep 09 20:12:03.250244 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.249924    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:12:03.292478 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.292440    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vm5kn"]
Sep 09 20:12:03.305755 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.305722    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4s4k"]
Sep 09 20:12:03.305935 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.305848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.308768 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.308744    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:12:03.308768 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.308759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5th2x\""
Sep 09 20:12:03.308768 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.308756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:12:03.314833 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.314796    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vm5kn"]
Sep 09 20:12:03.314959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.314855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4s4k"]
Sep 09 20:12:03.314959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.314936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.317119 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.317095    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b69kb"]
Sep 09 20:12:03.317838 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.317808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:12:03.317909 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.317841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xqpc7\""
Sep 09 20:12:03.317967 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.317955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:12:03.318006 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.317958    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:12:03.318067 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.318051    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:12:03.331524 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.331503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b69kb"]
Sep 09 20:12:03.331625 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.331587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.334159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.334142    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hmzcc\""
Sep 09 20:12:03.334280 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.334150    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:12:03.334280 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.334240    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:12:03.334424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.334409    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:12:03.385972 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.385934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5df9dd2b-eb41-4eea-bab2-927f81ac0386-config-volume\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.386140 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.385988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5df9dd2b-eb41-4eea-bab2-927f81ac0386-metrics-tls\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.386140 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.386052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfj8\" (UniqueName: \"kubernetes.io/projected/5df9dd2b-eb41-4eea-bab2-927f81ac0386-kube-api-access-pbfj8\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.386140 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.386114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df9dd2b-eb41-4eea-bab2-927f81ac0386-tmp-dir\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.420029 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.419996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:12:03.420174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.420111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:12:03.420174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.420165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:12:03.422789 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.422766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:03.422916 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.422768    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qbp9b\""
Sep 09 20:12:03.422978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.422926    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:03.423153 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.423138    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:12:03.423215 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.423201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8s569\""
Sep 09 20:12:03.423266 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.423221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:12:03.486744 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5df9dd2b-eb41-4eea-bab2-927f81ac0386-metrics-tls\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.486932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfj8\" (UniqueName: \"kubernetes.io/projected/5df9dd2b-eb41-4eea-bab2-927f81ac0386-kube-api-access-pbfj8\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.486932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88c270f0-4f8e-4d6f-861f-a49056c296b8-crio-socket\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.486932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df9dd2b-eb41-4eea-bab2-927f81ac0386-tmp-dir\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.486932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c270f0-4f8e-4d6f-861f-a49056c296b8-data-volume\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.486932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckk52\" (UniqueName: \"kubernetes.io/projected/da2252f6-c961-4a1d-b8b3-5923989737e9-kube-api-access-ckk52\") pod \"ingress-canary-b69kb\" (UID: \"da2252f6-c961-4a1d-b8b3-5923989737e9\") " pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.486932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c270f0-4f8e-4d6f-861f-a49056c296b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.487196 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.486945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2252f6-c961-4a1d-b8b3-5923989737e9-cert\") pod \"ingress-canary-b69kb\" (UID: \"da2252f6-c961-4a1d-b8b3-5923989737e9\") " pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.487196 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.487020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c270f0-4f8e-4d6f-861f-a49056c296b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.487196 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.487075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5df9dd2b-eb41-4eea-bab2-927f81ac0386-config-volume\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.487196 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.487105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hbk\" (UniqueName: \"kubernetes.io/projected/88c270f0-4f8e-4d6f-861f-a49056c296b8-kube-api-access-g5hbk\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.487322 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.487200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df9dd2b-eb41-4eea-bab2-927f81ac0386-tmp-dir\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.487533 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.487508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5df9dd2b-eb41-4eea-bab2-927f81ac0386-config-volume\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.491046 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.491005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5df9dd2b-eb41-4eea-bab2-927f81ac0386-metrics-tls\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.494408 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.494385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfj8\" (UniqueName: \"kubernetes.io/projected/5df9dd2b-eb41-4eea-bab2-927f81ac0386-kube-api-access-pbfj8\") pod \"dns-default-vm5kn\" (UID: \"5df9dd2b-eb41-4eea-bab2-927f81ac0386\") " pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.588159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hbk\" (UniqueName: \"kubernetes.io/projected/88c270f0-4f8e-4d6f-861f-a49056c296b8-kube-api-access-g5hbk\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.588413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88c270f0-4f8e-4d6f-861f-a49056c296b8-crio-socket\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.588413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c270f0-4f8e-4d6f-861f-a49056c296b8-data-volume\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.588413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckk52\" (UniqueName: \"kubernetes.io/projected/da2252f6-c961-4a1d-b8b3-5923989737e9-kube-api-access-ckk52\") pod \"ingress-canary-b69kb\" (UID: \"da2252f6-c961-4a1d-b8b3-5923989737e9\") " pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.588413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c270f0-4f8e-4d6f-861f-a49056c296b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.588413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2252f6-c961-4a1d-b8b3-5923989737e9-cert\") pod \"ingress-canary-b69kb\" (UID: \"da2252f6-c961-4a1d-b8b3-5923989737e9\") " pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.588413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c270f0-4f8e-4d6f-861f-a49056c296b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.588413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88c270f0-4f8e-4d6f-861f-a49056c296b8-crio-socket\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.588819 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.588684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c270f0-4f8e-4d6f-861f-a49056c296b8-data-volume\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.589851 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.589827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c270f0-4f8e-4d6f-861f-a49056c296b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.591075 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.591055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c270f0-4f8e-4d6f-861f-a49056c296b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.591460 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.591444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2252f6-c961-4a1d-b8b3-5923989737e9-cert\") pod \"ingress-canary-b69kb\" (UID: \"da2252f6-c961-4a1d-b8b3-5923989737e9\") " pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.609629 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.609606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckk52\" (UniqueName: \"kubernetes.io/projected/da2252f6-c961-4a1d-b8b3-5923989737e9-kube-api-access-ckk52\") pod \"ingress-canary-b69kb\" (UID: \"da2252f6-c961-4a1d-b8b3-5923989737e9\") " pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.609749 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.609610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hbk\" (UniqueName: \"kubernetes.io/projected/88c270f0-4f8e-4d6f-861f-a49056c296b8-kube-api-access-g5hbk\") pod \"insights-runtime-extractor-s4s4k\" (UID: \"88c270f0-4f8e-4d6f-861f-a49056c296b8\") " pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.619619 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.619598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:03.625753 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.625734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4s4k"
Sep 09 20:12:03.639861 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.639840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b69kb"
Sep 09 20:12:03.946721 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.946570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vm5kn"]
Sep 09 20:12:03.948952 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.948928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4s4k"]
Sep 09 20:12:03.951710 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:03.950459    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df9dd2b_eb41_4eea_bab2_927f81ac0386.slice/crio-60ee216e94f68b3bd07ac24737d1a4eb733bccc60e1d630fac0503e263db174b WatchSource:0}: Error finding container 60ee216e94f68b3bd07ac24737d1a4eb733bccc60e1d630fac0503e263db174b: Status 404 returned error can't find the container with id 60ee216e94f68b3bd07ac24737d1a4eb733bccc60e1d630fac0503e263db174b
Sep 09 20:12:03.953324 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:03.953303    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c270f0_4f8e_4d6f_861f_a49056c296b8.slice/crio-6d38bf44b6b06434396278e87f5c3e0cb2a8ba173de7828c699076d6fe97a765 WatchSource:0}: Error finding container 6d38bf44b6b06434396278e87f5c3e0cb2a8ba173de7828c699076d6fe97a765: Status 404 returned error can't find the container with id 6d38bf44b6b06434396278e87f5c3e0cb2a8ba173de7828c699076d6fe97a765
Sep 09 20:12:03.966596 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:03.966550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b69kb"]
Sep 09 20:12:03.970113 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:03.970091    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2252f6_c961_4a1d_b8b3_5923989737e9.slice/crio-c7b9c9f76163be23e496c0a79e617d1b14a426b67d505e2e3ae1d376db2fc45f WatchSource:0}: Error finding container c7b9c9f76163be23e496c0a79e617d1b14a426b67d505e2e3ae1d376db2fc45f: Status 404 returned error can't find the container with id c7b9c9f76163be23e496c0a79e617d1b14a426b67d505e2e3ae1d376db2fc45f
Sep 09 20:12:04.650006 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:04.649960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b69kb" event={"ID":"da2252f6-c961-4a1d-b8b3-5923989737e9","Type":"ContainerStarted","Data":"c7b9c9f76163be23e496c0a79e617d1b14a426b67d505e2e3ae1d376db2fc45f"}
Sep 09 20:12:04.651987 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:04.651947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4s4k" event={"ID":"88c270f0-4f8e-4d6f-861f-a49056c296b8","Type":"ContainerStarted","Data":"eb70cb780a1085e8048e49e21e5bf89219d141198bce3ddc38b56f5957278115"}
Sep 09 20:12:04.652115 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:04.652010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4s4k" event={"ID":"88c270f0-4f8e-4d6f-861f-a49056c296b8","Type":"ContainerStarted","Data":"6d38bf44b6b06434396278e87f5c3e0cb2a8ba173de7828c699076d6fe97a765"}
Sep 09 20:12:04.653529 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:04.653502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vm5kn" event={"ID":"5df9dd2b-eb41-4eea-bab2-927f81ac0386","Type":"ContainerStarted","Data":"60ee216e94f68b3bd07ac24737d1a4eb733bccc60e1d630fac0503e263db174b"}
Sep 09 20:12:06.428916 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.428885    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"]
Sep 09 20:12:06.431032 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.430997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"
Sep 09 20:12:06.433756 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.433728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 20:12:06.433890 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.433799    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-khdk4\""
Sep 09 20:12:06.442060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.442039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"]
Sep 09 20:12:06.613954 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.612563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8a94f10a-e0da-4cd7-9bb7-e4eda485d89a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-75cld\" (UID: \"8a94f10a-e0da-4cd7-9bb7-e4eda485d89a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"
Sep 09 20:12:06.660546 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.660520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b69kb" event={"ID":"da2252f6-c961-4a1d-b8b3-5923989737e9","Type":"ContainerStarted","Data":"10f4966b9dedc890a081b9f51ff6024ad5dd5f5213ecbb832af4caa2f3390936"}
Sep 09 20:12:06.663460 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.663437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4s4k" event={"ID":"88c270f0-4f8e-4d6f-861f-a49056c296b8","Type":"ContainerStarted","Data":"bae5ea755e1a9df0a938b8842053d3b5b2e2ea1eb44808fceef36a4625e302de"}
Sep 09 20:12:06.676679 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.676530    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b69kb" podStartSLOduration=1.066568637 podStartE2EDuration="3.676510991s" totalImagesPullingTime="2.609942354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:06.675570322 +0000 UTC m=+53.714740273" watchObservedRunningTime="2026-09-09 20:12:06.676510991 +0000 UTC m=+53.715680941"
Sep 09 20:12:06.713863 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.713831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8a94f10a-e0da-4cd7-9bb7-e4eda485d89a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-75cld\" (UID: \"8a94f10a-e0da-4cd7-9bb7-e4eda485d89a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"
Sep 09 20:12:06.718172 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.717783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8a94f10a-e0da-4cd7-9bb7-e4eda485d89a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-75cld\" (UID: \"8a94f10a-e0da-4cd7-9bb7-e4eda485d89a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"
Sep 09 20:12:06.740892 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:06.740863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"
Sep 09 20:12:07.020844 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:07.020767    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"]
Sep 09 20:12:07.024760 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:07.024721    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a94f10a_e0da_4cd7_9bb7_e4eda485d89a.slice/crio-f07ad0c01ae824ce2fd8576378e3db676403ef8fbaa3c764a061f15396ce3235 WatchSource:0}: Error finding container f07ad0c01ae824ce2fd8576378e3db676403ef8fbaa3c764a061f15396ce3235: Status 404 returned error can't find the container with id f07ad0c01ae824ce2fd8576378e3db676403ef8fbaa3c764a061f15396ce3235
Sep 09 20:12:07.667970 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:07.667933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld" event={"ID":"8a94f10a-e0da-4cd7-9bb7-e4eda485d89a","Type":"ContainerStarted","Data":"f07ad0c01ae824ce2fd8576378e3db676403ef8fbaa3c764a061f15396ce3235"}
Sep 09 20:12:07.670101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:07.670074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vm5kn" event={"ID":"5df9dd2b-eb41-4eea-bab2-927f81ac0386","Type":"ContainerStarted","Data":"573032cd8a0faedcd5e685c572e4c0571703d669bd09d37b0596c3334f901698"}
Sep 09 20:12:07.670217 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:07.670127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vm5kn" event={"ID":"5df9dd2b-eb41-4eea-bab2-927f81ac0386","Type":"ContainerStarted","Data":"e0247a7709b26d570cbc7106894f52ac6a30129a47bd5fcbaac36e8684469cf8"}
Sep 09 20:12:07.670285 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:07.670233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:07.690039 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:07.689977    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vm5kn" podStartSLOduration=2.065846393 podStartE2EDuration="4.689957552s" totalImagesPullingTime="2.624111159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:07.688622033 +0000 UTC m=+54.727791985" watchObservedRunningTime="2026-09-09 20:12:07.689957552 +0000 UTC m=+54.729127502"
Sep 09 20:12:08.675606 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:08.675571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4s4k" event={"ID":"88c270f0-4f8e-4d6f-861f-a49056c296b8","Type":"ContainerStarted","Data":"06de009d4970dd4b52ac9975ae90854f080e9a8efc2ad9676ec982ce9f897a3d"}
Sep 09 20:12:08.677146 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:08.677119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld" event={"ID":"8a94f10a-e0da-4cd7-9bb7-e4eda485d89a","Type":"ContainerStarted","Data":"5ea3cd80bdecac7b0539ca9108f60e3fb7bd879947e3a522c7918fb368e053a3"}
Sep 09 20:12:08.677323 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:08.677309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"
Sep 09 20:12:08.682042 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:08.682023    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld"
Sep 09 20:12:08.693559 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:08.693518    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s4s4k" podStartSLOduration=2.064762414 podStartE2EDuration="5.693504517s" totalImagesPullingTime="3.628742103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:08.691878827 +0000 UTC m=+55.731048777" watchObservedRunningTime="2026-09-09 20:12:08.693504517 +0000 UTC m=+55.732674462"
Sep 09 20:12:08.706592 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:08.706552    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-75cld" podStartSLOduration=1.154524323 podStartE2EDuration="2.706543522s" totalImagesPullingTime="1.552019199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:08.706057966 +0000 UTC m=+55.745227920" watchObservedRunningTime="2026-09-09 20:12:08.706543522 +0000 UTC m=+55.745713473"
Sep 09 20:12:09.491426 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.491396    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"]
Sep 09 20:12:09.493341 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.493327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.496093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.496065    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 20:12:09.496093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.496080    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:12:09.496271 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.496148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:12:09.497374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.497343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 20:12:09.497471 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.497384    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:12:09.497471 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.497416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-982pb\""
Sep 09 20:12:09.505095 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.505075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"]
Sep 09 20:12:09.635161 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.635119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.635329 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.635171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.635329 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.635249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgbk\" (UniqueName: \"kubernetes.io/projected/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-kube-api-access-tfgbk\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.635329 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.635271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.736413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.736375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgbk\" (UniqueName: \"kubernetes.io/projected/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-kube-api-access-tfgbk\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.736413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.736413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.736876 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.736447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.736876 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.736471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.737232 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.737212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.738941 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.738908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.738941 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.738933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.744909 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.744861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgbk\" (UniqueName: \"kubernetes.io/projected/3a61d7a5-ea84-4708-8fa8-8ea553254ab1-kube-api-access-tfgbk\") pod \"prometheus-operator-6f4946db56-cjx9l\" (UID: \"3a61d7a5-ea84-4708-8fa8-8ea553254ab1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:09.802176 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:09.802122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"
Sep 09 20:12:10.070794 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:10.070763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cjx9l"]
Sep 09 20:12:10.074907 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:10.074880    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a61d7a5_ea84_4708_8fa8_8ea553254ab1.slice/crio-173bcf99b0669c14d8c9b9978569dd299d42002c5e50a36eff97d040b2043932 WatchSource:0}: Error finding container 173bcf99b0669c14d8c9b9978569dd299d42002c5e50a36eff97d040b2043932: Status 404 returned error can't find the container with id 173bcf99b0669c14d8c9b9978569dd299d42002c5e50a36eff97d040b2043932
Sep 09 20:12:10.684220 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:10.684179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l" event={"ID":"3a61d7a5-ea84-4708-8fa8-8ea553254ab1","Type":"ContainerStarted","Data":"173bcf99b0669c14d8c9b9978569dd299d42002c5e50a36eff97d040b2043932"}
Sep 09 20:12:12.692358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:12.692324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l" event={"ID":"3a61d7a5-ea84-4708-8fa8-8ea553254ab1","Type":"ContainerStarted","Data":"21ca8fc630505165553cf288be4a5fef965b607832f68a36bebe9d7e81f57fd5"}
Sep 09 20:12:12.692757 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:12.692375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l" event={"ID":"3a61d7a5-ea84-4708-8fa8-8ea553254ab1","Type":"ContainerStarted","Data":"43aac77b553bcecbe18dcd617a508270f4dc5244e8e080cc95c01d8f00eb82fa"}
Sep 09 20:12:12.710124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:12.710072    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-cjx9l" podStartSLOduration=1.865670233 podStartE2EDuration="3.710056697s" totalImagesPullingTime="1.844386464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:12.708628189 +0000 UTC m=+59.747798140" watchObservedRunningTime="2026-09-09 20:12:12.710056697 +0000 UTC m=+59.749226647"
Sep 09 20:12:13.595626 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:13.595596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9fx7"
Sep 09 20:12:14.151844 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.151813    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-v7dx9"]
Sep 09 20:12:14.154475 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.154456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-v7dx9"
Sep 09 20:12:14.157258 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.157230    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-smmh9\""
Sep 09 20:12:14.157383 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.157276    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:12:14.157383 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.157276    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:12:14.164630 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.164611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-v7dx9"]
Sep 09 20:12:14.270090 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.270052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r28vl\" (UniqueName: \"kubernetes.io/projected/552cc6ef-0359-43f2-9e11-7706454be021-kube-api-access-r28vl\") pod \"downloads-85d98c998f-v7dx9\" (UID: \"552cc6ef-0359-43f2-9e11-7706454be021\") " pod="openshift-console/downloads-85d98c998f-v7dx9"
Sep 09 20:12:14.370433 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.370392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r28vl\" (UniqueName: \"kubernetes.io/projected/552cc6ef-0359-43f2-9e11-7706454be021-kube-api-access-r28vl\") pod \"downloads-85d98c998f-v7dx9\" (UID: \"552cc6ef-0359-43f2-9e11-7706454be021\") " pod="openshift-console/downloads-85d98c998f-v7dx9"
Sep 09 20:12:14.380283 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.380263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r28vl\" (UniqueName: \"kubernetes.io/projected/552cc6ef-0359-43f2-9e11-7706454be021-kube-api-access-r28vl\") pod \"downloads-85d98c998f-v7dx9\" (UID: \"552cc6ef-0359-43f2-9e11-7706454be021\") " pod="openshift-console/downloads-85d98c998f-v7dx9"
Sep 09 20:12:14.463478 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.463385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-v7dx9"
Sep 09 20:12:14.731785 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.731709    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-v7dx9"]
Sep 09 20:12:14.736242 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:14.736208    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552cc6ef_0359_43f2_9e11_7706454be021.slice/crio-65eb4eaa346ce511dc4d096756ee10b71065c294d85d4d7d05eea342ecde8ff3 WatchSource:0}: Error finding container 65eb4eaa346ce511dc4d096756ee10b71065c294d85d4d7d05eea342ecde8ff3: Status 404 returned error can't find the container with id 65eb4eaa346ce511dc4d096756ee10b71065c294d85d4d7d05eea342ecde8ff3
Sep 09 20:12:14.875143 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.875110    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zgpc8"]
Sep 09 20:12:14.878124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.878101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.880977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.880954    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:12:14.880977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.880966    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:12:14.881150 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.880958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:12:14.881150 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.880958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sm7xj\""
Sep 09 20:12:14.976026 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.975994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-accelerators-collector-config\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976193 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-sys\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976193 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-root\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976193 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-textfile\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-tls\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-wtmp\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a22e518a-a8bf-4962-8c2f-1223a20c2114-metrics-client-ca\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:14.976342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:14.976281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfx7\" (UniqueName: \"kubernetes.io/projected/a22e518a-a8bf-4962-8c2f-1223a20c2114-kube-api-access-hwfx7\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077456 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-sys\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077456 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-root\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-textfile\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-tls\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-sys\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-root\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-wtmp\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a22e518a-a8bf-4962-8c2f-1223a20c2114-metrics-client-ca\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-wtmp\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfx7\" (UniqueName: \"kubernetes.io/projected/a22e518a-a8bf-4962-8c2f-1223a20c2114-kube-api-access-hwfx7\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-accelerators-collector-config\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.077966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.077922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-textfile\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.078304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.078272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a22e518a-a8bf-4962-8c2f-1223a20c2114-metrics-client-ca\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.078386 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.078306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-accelerators-collector-config\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.080228 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.080209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-tls\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.080762 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.080734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a22e518a-a8bf-4962-8c2f-1223a20c2114-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.086172 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.086154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfx7\" (UniqueName: \"kubernetes.io/projected/a22e518a-a8bf-4962-8c2f-1223a20c2114-kube-api-access-hwfx7\") pod \"node-exporter-zgpc8\" (UID: \"a22e518a-a8bf-4962-8c2f-1223a20c2114\") " pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.186891 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.186853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zgpc8"
Sep 09 20:12:15.196810 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:15.196783    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22e518a_a8bf_4962_8c2f_1223a20c2114.slice/crio-d86321398e15a9f23f0120b8b1347d7e11c19175240377c94fc3bc8d3e8ebfa1 WatchSource:0}: Error finding container d86321398e15a9f23f0120b8b1347d7e11c19175240377c94fc3bc8d3e8ebfa1: Status 404 returned error can't find the container with id d86321398e15a9f23f0120b8b1347d7e11c19175240377c94fc3bc8d3e8ebfa1
Sep 09 20:12:15.705735 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.705686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgpc8" event={"ID":"a22e518a-a8bf-4962-8c2f-1223a20c2114","Type":"ContainerStarted","Data":"d86321398e15a9f23f0120b8b1347d7e11c19175240377c94fc3bc8d3e8ebfa1"}
Sep 09 20:12:15.707137 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.707099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-v7dx9" event={"ID":"552cc6ef-0359-43f2-9e11-7706454be021","Type":"ContainerStarted","Data":"65eb4eaa346ce511dc4d096756ee10b71065c294d85d4d7d05eea342ecde8ff3"}
Sep 09 20:12:15.929210 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.929174    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:12:15.935190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.935158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.939203 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.939168    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:12:15.939442 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.939420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:12:15.940683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.940034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:12:15.940683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.940070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:12:15.940683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.940078    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:12:15.940683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.940268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:12:15.940683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.940297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:12:15.942125 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.941070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9frg8\""
Sep 09 20:12:15.942125 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.941528    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:12:15.942125 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.941853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:12:15.957106 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.957038    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:12:15.987485 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-volume\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvk5q\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-kube-api-access-wvk5q\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-web-config\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-out\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:15.987978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:15.987952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088507 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088718 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-out\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088718 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088718 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088718 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088718 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:12:16.088713    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle podName:ccfe2555-e18f-4bc0-b5c0-8a71e23f5400 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:16.588687044 +0000 UTC m=+63.627856992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400") : configmap references non-existent config key: ca-bundle.crt
Sep 09 20:12:16.088977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-volume\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.088977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvk5q\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-kube-api-access-wvk5q\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.089262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.088996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-web-config\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.089262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.089053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.089531 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.089280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.089531 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:12:16.089396    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 20:12:16.089531 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.089405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.089531 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:12:16.089476    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls podName:ccfe2555-e18f-4bc0-b5c0-8a71e23f5400 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:16.589443988 +0000 UTC m=+63.628613940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400") : secret "alertmanager-main-tls" not found
Sep 09 20:12:16.091928 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.091860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.091928 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.091933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.092460 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.092264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-out\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.092585 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.092565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-web-config\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.093771 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.093699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.093976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.093932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.094245 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.094223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-volume\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.094302 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.094279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.101705 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.101680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvk5q\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-kube-api-access-wvk5q\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.592312 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.592269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.592810 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.592402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.593397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.593366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.595231 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.595207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:16.715310 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.715278    2563 generic.go:356] "Generic (PLEG): container finished" podID="a22e518a-a8bf-4962-8c2f-1223a20c2114" containerID="7dab0a0865f3091d24b6dcf917c3fc1d80fa6235127077d3d0e67f5362f43efe" exitCode=0
Sep 09 20:12:16.715459 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.715374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgpc8" event={"ID":"a22e518a-a8bf-4962-8c2f-1223a20c2114","Type":"ContainerDied","Data":"7dab0a0865f3091d24b6dcf917c3fc1d80fa6235127077d3d0e67f5362f43efe"}
Sep 09 20:12:16.850062 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:16.850002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:17.142787 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.142762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:12:17.145629 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:17.145602    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfe2555_e18f_4bc0_b5c0_8a71e23f5400.slice/crio-77d2fc75d2956c7dd8266167b3d1b14cd11ab7ce8a5ed9e0057ee027c8846186 WatchSource:0}: Error finding container 77d2fc75d2956c7dd8266167b3d1b14cd11ab7ce8a5ed9e0057ee027c8846186: Status 404 returned error can't find the container with id 77d2fc75d2956c7dd8266167b3d1b14cd11ab7ce8a5ed9e0057ee027c8846186
Sep 09 20:12:17.680201 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.680166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vm5kn"
Sep 09 20:12:17.723335 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.723298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgpc8" event={"ID":"a22e518a-a8bf-4962-8c2f-1223a20c2114","Type":"ContainerStarted","Data":"87e7a5207962bdb9aa5d6ec575f83c92c16cd93b0a1fe2ec05bdfa6666af3ba9"}
Sep 09 20:12:17.723503 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.723361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zgpc8" event={"ID":"a22e518a-a8bf-4962-8c2f-1223a20c2114","Type":"ContainerStarted","Data":"03a3e4f22d756a16f30fe4b55af555b2d247da97e4067848dc8b9c1d77d5a32c"}
Sep 09 20:12:17.725270 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.725229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"77d2fc75d2956c7dd8266167b3d1b14cd11ab7ce8a5ed9e0057ee027c8846186"}
Sep 09 20:12:17.746918 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.746405    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zgpc8" podStartSLOduration=2.566690773 podStartE2EDuration="3.746383353s" totalImagesPullingTime="1.100278795s" totalInitContainerRuntime="79.413785ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:17.746240713 +0000 UTC m=+64.785410665" watchObservedRunningTime="2026-09-09 20:12:17.746383353 +0000 UTC m=+64.785553304"
Sep 09 20:12:17.831412 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.831375    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5684db94c7-rt24t"]
Sep 09 20:12:17.834468 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.834442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.837375 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.837190    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 20:12:17.840222 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.840195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 20:12:17.840994 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.840339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 20:12:17.840994 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.840464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 20:12:17.840994 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.840567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-frlecrc1vs65r\""
Sep 09 20:12:17.840994 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.840699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 20:12:17.840994 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.840797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4dds6\""
Sep 09 20:12:17.846335 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.846308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5684db94c7-rt24t"]
Sep 09 20:12:17.901940 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.901905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-metrics-client-ca\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.902108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.901955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.902108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.902005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-tls\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.902108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.902030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-grpc-tls\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.902108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.902064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22g5\" (UniqueName: \"kubernetes.io/projected/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-kube-api-access-x22g5\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.902108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.902081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.902108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.902101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:17.902439 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:17.902150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003464 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003464 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-metrics-client-ca\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003732 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003732 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-tls\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003732 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-grpc-tls\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003732 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22g5\" (UniqueName: \"kubernetes.io/projected/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-kube-api-access-x22g5\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003732 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.003732 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.003686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.005376 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.005327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-metrics-client-ca\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.007130 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.007108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.007238 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.007206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-tls\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.007703 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.007664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.009421 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.009398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-grpc-tls\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.012124 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.012099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.013112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.013088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22g5\" (UniqueName: \"kubernetes.io/projected/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-kube-api-access-x22g5\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.024544 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.024266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bae5fbac-03ce-4e1a-8667-e6042edd1b5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5684db94c7-rt24t\" (UID: \"bae5fbac-03ce-4e1a-8667-e6042edd1b5f\") " pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.150080 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.149817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:18.560822 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.560763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5684db94c7-rt24t"]
Sep 09 20:12:18.581904 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:18.581859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae5fbac_03ce_4e1a_8667_e6042edd1b5f.slice/crio-a31879b327b857047370b6749a6921e4065984924c7abddc697c556c7b33628f WatchSource:0}: Error finding container a31879b327b857047370b6749a6921e4065984924c7abddc697c556c7b33628f: Status 404 returned error can't find the container with id a31879b327b857047370b6749a6921e4065984924c7abddc697c556c7b33628f
Sep 09 20:12:18.731231 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.731193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32"}
Sep 09 20:12:18.732719 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:18.732688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" event={"ID":"bae5fbac-03ce-4e1a-8667-e6042edd1b5f","Type":"ContainerStarted","Data":"a31879b327b857047370b6749a6921e4065984924c7abddc697c556c7b33628f"}
Sep 09 20:12:19.114413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.114371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:12:19.116974 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.116941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:12:19.127538 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.127509    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:12:19.138084 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.138059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj587\" (UniqueName: \"kubernetes.io/projected/3dcd3207-d2b6-4acb-8a11-5dffd25b0adf-kube-api-access-qj587\") pod \"network-check-target-855rx\" (UID: \"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf\") " pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:12:19.151413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.151380    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56cf9559d6-hk8xp"]
Sep 09 20:12:19.176057 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.176032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56cf9559d6-hk8xp"]
Sep 09 20:12:19.176190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.176141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.181276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.180478    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cxkhh\""
Sep 09 20:12:19.181276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.180527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3nrpmu4oarv8o\""
Sep 09 20:12:19.181276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.180563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 20:12:19.181276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.180564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 20:12:19.181276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.180633    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 20:12:19.181276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.180711    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 20:12:19.215616 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf95e321-da7d-4b72-8e98-5bd03a0b2176-audit-log\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.215616 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf95e321-da7d-4b72-8e98-5bd03a0b2176-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.215856 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66b84\" (UniqueName: \"kubernetes.io/projected/cf95e321-da7d-4b72-8e98-5bd03a0b2176-kube-api-access-66b84\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.215856 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-client-ca-bundle\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.215856 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-secret-metrics-server-tls\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.215856 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf95e321-da7d-4b72-8e98-5bd03a0b2176-metrics-server-audit-profiles\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.215856 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:12:19.216102 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-secret-metrics-server-client-certs\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.216102 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.215926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:12:19.218724 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.218699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:19.218839 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.218700    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:19.228905 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.228876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e68f0176-5891-423d-ae93-8c6c8cdf1d93-original-pull-secret\") pod \"global-pull-secret-syncer-gz6b2\" (UID: \"e68f0176-5891-423d-ae93-8c6c8cdf1d93\") " pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:12:19.229030 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.228974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c6873b-0047-43fc-a158-8ff4ceb8e0dd-metrics-certs\") pod \"network-metrics-daemon-jklcc\" (UID: \"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd\") " pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:12:19.316785 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.316751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf95e321-da7d-4b72-8e98-5bd03a0b2176-audit-log\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.316976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.316795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf95e321-da7d-4b72-8e98-5bd03a0b2176-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.316976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.316827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66b84\" (UniqueName: \"kubernetes.io/projected/cf95e321-da7d-4b72-8e98-5bd03a0b2176-kube-api-access-66b84\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.316976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.316861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-client-ca-bundle\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.316976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.316882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-secret-metrics-server-tls\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.316976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.316910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf95e321-da7d-4b72-8e98-5bd03a0b2176-metrics-server-audit-profiles\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.316976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.316944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-secret-metrics-server-client-certs\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.317273 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.317217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf95e321-da7d-4b72-8e98-5bd03a0b2176-audit-log\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.318763 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.317644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf95e321-da7d-4b72-8e98-5bd03a0b2176-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.318763 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.318720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf95e321-da7d-4b72-8e98-5bd03a0b2176-metrics-server-audit-profiles\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.319942 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.319920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-secret-metrics-server-tls\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.320053 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.320035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-client-ca-bundle\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.320093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.320067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf95e321-da7d-4b72-8e98-5bd03a0b2176-secret-metrics-server-client-certs\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.325903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.325883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66b84\" (UniqueName: \"kubernetes.io/projected/cf95e321-da7d-4b72-8e98-5bd03a0b2176-kube-api-access-66b84\") pod \"metrics-server-56cf9559d6-hk8xp\" (UID: \"cf95e321-da7d-4b72-8e98-5bd03a0b2176\") " pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.332240 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.332219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qbp9b\""
Sep 09 20:12:19.335536 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.335515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz6b2"
Sep 09 20:12:19.339706 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.339686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:12:19.342154 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.342133    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8s569\""
Sep 09 20:12:19.350260 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.350242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jklcc"
Sep 09 20:12:19.486340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.486043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:19.634123 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.632928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gz6b2"]
Sep 09 20:12:19.636257 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:19.636229    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68f0176_5891_423d_ae93_8c6c8cdf1d93.slice/crio-c0d0a842e44660e30ae6cde16029148e8178a2566606134fd82a7c5ebff8918c WatchSource:0}: Error finding container c0d0a842e44660e30ae6cde16029148e8178a2566606134fd82a7c5ebff8918c: Status 404 returned error can't find the container with id c0d0a842e44660e30ae6cde16029148e8178a2566606134fd82a7c5ebff8918c
Sep 09 20:12:19.683834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.683798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-855rx"]
Sep 09 20:12:19.686510 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:19.686473    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcd3207_d2b6_4acb_8a11_5dffd25b0adf.slice/crio-a56838de4a94d07339e3efff5786e6070d70317ca0c0b64a827153b0c980d9f1 WatchSource:0}: Error finding container a56838de4a94d07339e3efff5786e6070d70317ca0c0b64a827153b0c980d9f1: Status 404 returned error can't find the container with id a56838de4a94d07339e3efff5786e6070d70317ca0c0b64a827153b0c980d9f1
Sep 09 20:12:19.712271 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.712241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jklcc"]
Sep 09 20:12:19.720791 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:19.720760    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c6873b_0047_43fc_a158_8ff4ceb8e0dd.slice/crio-7ab19cd1674d6d736e486ec56a286e43a661f76ab821434400fda045d8091fb1 WatchSource:0}: Error finding container 7ab19cd1674d6d736e486ec56a286e43a661f76ab821434400fda045d8091fb1: Status 404 returned error can't find the container with id 7ab19cd1674d6d736e486ec56a286e43a661f76ab821434400fda045d8091fb1
Sep 09 20:12:19.739126 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.739095    2563 generic.go:356] "Generic (PLEG): container finished" podID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerID="66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32" exitCode=0
Sep 09 20:12:19.739558 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.739167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32"}
Sep 09 20:12:19.745142 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.741517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jklcc" event={"ID":"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd","Type":"ContainerStarted","Data":"7ab19cd1674d6d736e486ec56a286e43a661f76ab821434400fda045d8091fb1"}
Sep 09 20:12:19.745415 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.745384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-855rx" event={"ID":"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf","Type":"ContainerStarted","Data":"a56838de4a94d07339e3efff5786e6070d70317ca0c0b64a827153b0c980d9f1"}
Sep 09 20:12:19.747451 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.747426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gz6b2" event={"ID":"e68f0176-5891-423d-ae93-8c6c8cdf1d93","Type":"ContainerStarted","Data":"c0d0a842e44660e30ae6cde16029148e8178a2566606134fd82a7c5ebff8918c"}
Sep 09 20:12:19.810114 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:19.809888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56cf9559d6-hk8xp"]
Sep 09 20:12:19.814722 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:19.814690    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf95e321_da7d_4b72_8e98_5bd03a0b2176.slice/crio-904e2fcd0f72facc9da5dbfceb3b286ac955e8f498bbe44c01c4a8ace3ead56b WatchSource:0}: Error finding container 904e2fcd0f72facc9da5dbfceb3b286ac955e8f498bbe44c01c4a8ace3ead56b: Status 404 returned error can't find the container with id 904e2fcd0f72facc9da5dbfceb3b286ac955e8f498bbe44c01c4a8ace3ead56b
Sep 09 20:12:20.758251 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:20.758207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp" event={"ID":"cf95e321-da7d-4b72-8e98-5bd03a0b2176","Type":"ContainerStarted","Data":"904e2fcd0f72facc9da5dbfceb3b286ac955e8f498bbe44c01c4a8ace3ead56b"}
Sep 09 20:12:24.449483 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.449443    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54cc987686-46gkn"]
Sep 09 20:12:24.452937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.451996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.455111 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.455062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:12:24.455111 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.455087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:12:24.455277 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.455062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:12:24.456291 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.456270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:12:24.456291 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.456285    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:12:24.456429 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.456401    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r4hrd\""
Sep 09 20:12:24.463090 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.463066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cc987686-46gkn"]
Sep 09 20:12:24.579272 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.579242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-serving-cert\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.579272 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.579277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-oauth-config\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.579493 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.579308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-console-config\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.579493 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.579363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-oauth-serving-cert\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.579493 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.579455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-service-ca\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.579674 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.579513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cph7\" (UniqueName: \"kubernetes.io/projected/ab583534-480a-46ac-bfce-f12c3b0c3098-kube-api-access-7cph7\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.680117 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.680079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cph7\" (UniqueName: \"kubernetes.io/projected/ab583534-480a-46ac-bfce-f12c3b0c3098-kube-api-access-7cph7\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.680299 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.680159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-serving-cert\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.680299 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.680190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-oauth-config\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.680299 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.680228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-console-config\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.680299 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.680251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-oauth-serving-cert\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.680299 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.680282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-service-ca\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.681165 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.681140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-service-ca\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.682481 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.682451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-console-config\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.682585 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.682543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-oauth-serving-cert\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.684353 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.684324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-serving-cert\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.684673 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.684626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-oauth-config\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.690119 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.690098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cph7\" (UniqueName: \"kubernetes.io/projected/ab583534-480a-46ac-bfce-f12c3b0c3098-kube-api-access-7cph7\") pod \"console-54cc987686-46gkn\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") " pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:24.764135 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:24.764046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:26.632924 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.632894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cc987686-46gkn"]
Sep 09 20:12:26.790173 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.790136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0"}
Sep 09 20:12:26.790338 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.790218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93"}
Sep 09 20:12:26.790338 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.790259    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b"}
Sep 09 20:12:26.792637 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.792607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jklcc" event={"ID":"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd","Type":"ContainerStarted","Data":"ce1077057ddcc51a6ce6ac6a14fa453897e4e62d4cd11ea6f43e8c536ed1751f"}
Sep 09 20:12:26.792786 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.792678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jklcc" event={"ID":"a5c6873b-0047-43fc-a158-8ff4ceb8e0dd","Type":"ContainerStarted","Data":"1e8a93c3edbbbbebf2d7ad0d0880674835f2de0f28fc51174b67d99f144c1747"}
Sep 09 20:12:26.795387 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.795363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-855rx" event={"ID":"3dcd3207-d2b6-4acb-8a11-5dffd25b0adf","Type":"ContainerStarted","Data":"f34d9dbf4c7bced6c23af0fd5cd819853a6a66e3d18da9b04abdec99c5185c37"}
Sep 09 20:12:26.795868 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.795849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:12:26.799550 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.799078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp" event={"ID":"cf95e321-da7d-4b72-8e98-5bd03a0b2176","Type":"ContainerStarted","Data":"f7c1de8312c2552620f104b12be8a60217ad8be85eb64ce326f0c51bfb9bba75"}
Sep 09 20:12:26.802388 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.802216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" event={"ID":"bae5fbac-03ce-4e1a-8667-e6042edd1b5f","Type":"ContainerStarted","Data":"843cba657f2c44bc931f8814d33c4f732dddef1039b597253c77af03ae579767"}
Sep 09 20:12:26.802388 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.802282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" event={"ID":"bae5fbac-03ce-4e1a-8667-e6042edd1b5f","Type":"ContainerStarted","Data":"0ea7c78ad097391bebaff65037f2e3cca1ab147718f17a940a5d4af9a05c03cc"}
Sep 09 20:12:26.802388 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.802327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" event={"ID":"bae5fbac-03ce-4e1a-8667-e6042edd1b5f","Type":"ContainerStarted","Data":"6d14cf82f66533ed9a6b0bbfc130d530a4c9f9067ee757ca89dde23966b67dd8"}
Sep 09 20:12:26.804642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.804208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gz6b2" event={"ID":"e68f0176-5891-423d-ae93-8c6c8cdf1d93","Type":"ContainerStarted","Data":"0bf4a675a3006cc8b9c30ed8f6d86b8f6aa35015bb2a042306b9541bc6aedf45"}
Sep 09 20:12:26.806069 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.806023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc987686-46gkn" event={"ID":"ab583534-480a-46ac-bfce-f12c3b0c3098","Type":"ContainerStarted","Data":"ca6dc718d87ed29488a9a8119f620be7b330e1e545eb18f84852b7c2db481871"}
Sep 09 20:12:26.810170 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.809836    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jklcc" podStartSLOduration=67.202414579 podStartE2EDuration="1m13.809818634s" totalImagesPullingTime="6.607404055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.809185069 +0000 UTC m=+73.848355017" watchObservedRunningTime="2026-09-09 20:12:26.809818634 +0000 UTC m=+73.848988586"
Sep 09 20:12:26.823979 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.823362    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-855rx" podStartSLOduration=67.157600765 podStartE2EDuration="1m13.823343806s" totalImagesPullingTime="6.665743041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.822873068 +0000 UTC m=+73.862043018" watchObservedRunningTime="2026-09-09 20:12:26.823343806 +0000 UTC m=+73.862513757"
Sep 09 20:12:26.838474 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.838379    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gz6b2" podStartSLOduration=67.10086636 podStartE2EDuration="1m13.838360219s" totalImagesPullingTime="6.737493859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.837288528 +0000 UTC m=+73.876458479" watchObservedRunningTime="2026-09-09 20:12:26.838360219 +0000 UTC m=+73.877530172"
Sep 09 20:12:26.856620 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:26.856563    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp" podStartSLOduration=1.342150835 podStartE2EDuration="7.856541208s" totalImagesPullingTime="6.514390373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.854597468 +0000 UTC m=+73.893767434" watchObservedRunningTime="2026-09-09 20:12:26.856541208 +0000 UTC m=+73.895711162"
Sep 09 20:12:33.864528 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.864496    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f99ff8bb-x7gst"]
Sep 09 20:12:33.902071 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.902036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f99ff8bb-x7gst"]
Sep 09 20:12:33.902229 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.902213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:33.914054 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.914026    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:12:33.974005 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.973970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-serving-cert\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:33.974005 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.974004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-oauth-config\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:33.974217 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.974022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-trusted-ca-bundle\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:33.974217 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.974083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-config\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:33.974217 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.974162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkt7t\" (UniqueName: \"kubernetes.io/projected/3eb19115-e909-4d75-8984-8bd247b2e8b7-kube-api-access-pkt7t\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:33.974217 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.974201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-service-ca\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:33.974217 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:33.974215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-oauth-serving-cert\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.074983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.074956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-config\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.075080 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.074997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkt7t\" (UniqueName: \"kubernetes.io/projected/3eb19115-e909-4d75-8984-8bd247b2e8b7-kube-api-access-pkt7t\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.075080 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.075044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-service-ca\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.075080 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.075070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-oauth-serving-cert\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.075222 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.075135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-serving-cert\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.075222 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.075160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-oauth-config\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.075222 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.075189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-trusted-ca-bundle\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.076353 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.076328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-oauth-serving-cert\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.076465 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.076139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-service-ca\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.076465 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.076328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-trusted-ca-bundle\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.076465 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.076382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-config\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.077968 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.077945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-oauth-config\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.078097 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.078080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-serving-cert\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.082874 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.082852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkt7t\" (UniqueName: \"kubernetes.io/projected/3eb19115-e909-4d75-8984-8bd247b2e8b7-kube-api-access-pkt7t\") pod \"console-5f99ff8bb-x7gst\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") " pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.216882 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.216855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:34.654398 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.654322    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f99ff8bb-x7gst"]
Sep 09 20:12:34.701676 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:12:34.701570    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb19115_e909_4d75_8984_8bd247b2e8b7.slice/crio-5f71fb4d59e4b6e65cef3bde745c970b170254649119014df283c4df66db7d35 WatchSource:0}: Error finding container 5f71fb4d59e4b6e65cef3bde745c970b170254649119014df283c4df66db7d35: Status 404 returned error can't find the container with id 5f71fb4d59e4b6e65cef3bde745c970b170254649119014df283c4df66db7d35
Sep 09 20:12:34.850818 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.850642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b"}
Sep 09 20:12:34.850818 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.850752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d"}
Sep 09 20:12:34.853987 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.853075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-v7dx9" event={"ID":"552cc6ef-0359-43f2-9e11-7706454be021","Type":"ContainerStarted","Data":"8d08cd8d8d5a90a5c810afd3560c4fbba1811a7dacae1cd6b8910d6ca7e6e863"}
Sep 09 20:12:34.853987 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.853946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-v7dx9"
Sep 09 20:12:34.857390 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.856730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" event={"ID":"bae5fbac-03ce-4e1a-8667-e6042edd1b5f","Type":"ContainerStarted","Data":"5441b0ac54c276727b341ddf7f0c3f98466682417c45a1727dfeed0985cc8dcc"}
Sep 09 20:12:34.858433 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.858394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-v7dx9"
Sep 09 20:12:34.859260 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.859197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f99ff8bb-x7gst" event={"ID":"3eb19115-e909-4d75-8984-8bd247b2e8b7","Type":"ContainerStarted","Data":"5f71fb4d59e4b6e65cef3bde745c970b170254649119014df283c4df66db7d35"}
Sep 09 20:12:34.871787 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:34.869742    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-v7dx9" podStartSLOduration=1.171362011 podStartE2EDuration="20.869723403s" totalImagesPullingTime="19.698361392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:34.868497741 +0000 UTC m=+81.907667692" watchObservedRunningTime="2026-09-09 20:12:34.869723403 +0000 UTC m=+81.908893352"
Sep 09 20:12:35.887675 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:35.886711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" event={"ID":"bae5fbac-03ce-4e1a-8667-e6042edd1b5f","Type":"ContainerStarted","Data":"efc0555f3339d3effd3238c3a235dbfc6ed3c0caa254376fd9eeea999cf0df2a"}
Sep 09 20:12:35.887675 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:35.886830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" event={"ID":"bae5fbac-03ce-4e1a-8667-e6042edd1b5f","Type":"ContainerStarted","Data":"967f04963f16d76d71d82fd1d55e16767c0109c6e6b3ebec60fb205d73ae59d3"}
Sep 09 20:12:35.888282 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:35.888248    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:35.895308 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:35.895286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t"
Sep 09 20:12:35.897540 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:35.897518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerStarted","Data":"4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a"}
Sep 09 20:12:35.920108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:35.920027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5684db94c7-rt24t" podStartSLOduration=3.158595242 podStartE2EDuration="18.920006451s" totalImagesPullingTime="15.761411209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:35.917052423 +0000 UTC m=+82.956222365" watchObservedRunningTime="2026-09-09 20:12:35.920006451 +0000 UTC m=+82.959176403"
Sep 09 20:12:35.967273 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:35.966828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=12.412660234 podStartE2EDuration="20.966802369s" totalImagesPullingTime="8.52716295s" totalInitContainerRuntime="26.979185ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:35.963550282 +0000 UTC m=+83.002720235" watchObservedRunningTime="2026-09-09 20:12:35.966802369 +0000 UTC m=+83.005972321"
Sep 09 20:12:39.487262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:39.487224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:39.487739 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:39.487309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:39.920716 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:39.920677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc987686-46gkn" event={"ID":"ab583534-480a-46ac-bfce-f12c3b0c3098","Type":"ContainerStarted","Data":"994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545"}
Sep 09 20:12:39.922748 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:39.922715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f99ff8bb-x7gst" event={"ID":"3eb19115-e909-4d75-8984-8bd247b2e8b7","Type":"ContainerStarted","Data":"21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a"}
Sep 09 20:12:39.941736 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:39.941682    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54cc987686-46gkn" podStartSLOduration=3.775028138 podStartE2EDuration="15.941666862s" totalImagesPullingTime="12.166638724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:39.93930949 +0000 UTC m=+86.978479441" watchObservedRunningTime="2026-09-09 20:12:39.941666862 +0000 UTC m=+86.980836808"
Sep 09 20:12:39.958695 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:39.958630    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f99ff8bb-x7gst" podStartSLOduration=2.847082037 podStartE2EDuration="6.958612681s" totalImagesPullingTime="4.111530644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:39.957170581 +0000 UTC m=+86.996340532" watchObservedRunningTime="2026-09-09 20:12:39.958612681 +0000 UTC m=+86.997782632"
Sep 09 20:12:44.217290 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.217254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:44.217774 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.217336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:44.222069 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.222047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:44.764782 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.764749    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:44.764948 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.764809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:44.769417 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.769398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:44.948509 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.948481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:12:44.948992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:44.948971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:12:45.012174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:45.012146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cc987686-46gkn"]
Sep 09 20:12:58.821358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:58.821317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-855rx"
Sep 09 20:12:59.492434 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:59.492401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:12:59.496134 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:12:59.496109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56cf9559d6-hk8xp"
Sep 09 20:13:11.973430 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:11.973374    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54cc987686-46gkn" podUID="ab583534-480a-46ac-bfce-f12c3b0c3098" containerName="console" containerID="cri-o://994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545" gracePeriod=15
Sep 09 20:13:12.251242 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.251173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:13:12.295453 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-oauth-config\") pod \"ab583534-480a-46ac-bfce-f12c3b0c3098\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") "
Sep 09 20:13:12.295453 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-oauth-serving-cert\") pod \"ab583534-480a-46ac-bfce-f12c3b0c3098\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") "
Sep 09 20:13:12.295709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295481    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cph7\" (UniqueName: \"kubernetes.io/projected/ab583534-480a-46ac-bfce-f12c3b0c3098-kube-api-access-7cph7\") pod \"ab583534-480a-46ac-bfce-f12c3b0c3098\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") "
Sep 09 20:13:12.295709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295522    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-service-ca\") pod \"ab583534-480a-46ac-bfce-f12c3b0c3098\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") "
Sep 09 20:13:12.295709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-console-config\") pod \"ab583534-480a-46ac-bfce-f12c3b0c3098\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") "
Sep 09 20:13:12.295709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-serving-cert\") pod \"ab583534-480a-46ac-bfce-f12c3b0c3098\" (UID: \"ab583534-480a-46ac-bfce-f12c3b0c3098\") "
Sep 09 20:13:12.296001 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ab583534-480a-46ac-bfce-f12c3b0c3098" (UID: "ab583534-480a-46ac-bfce-f12c3b0c3098"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:12.296090 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.295997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-service-ca" (OuterVolumeSpecName: "service-ca") pod "ab583534-480a-46ac-bfce-f12c3b0c3098" (UID: "ab583534-480a-46ac-bfce-f12c3b0c3098"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:12.296090 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.296045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-console-config" (OuterVolumeSpecName: "console-config") pod "ab583534-480a-46ac-bfce-f12c3b0c3098" (UID: "ab583534-480a-46ac-bfce-f12c3b0c3098"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:12.298072 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.298040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ab583534-480a-46ac-bfce-f12c3b0c3098" (UID: "ab583534-480a-46ac-bfce-f12c3b0c3098"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:12.298163 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.298116    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ab583534-480a-46ac-bfce-f12c3b0c3098" (UID: "ab583534-480a-46ac-bfce-f12c3b0c3098"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:12.298163 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.298122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab583534-480a-46ac-bfce-f12c3b0c3098-kube-api-access-7cph7" (OuterVolumeSpecName: "kube-api-access-7cph7") pod "ab583534-480a-46ac-bfce-f12c3b0c3098" (UID: "ab583534-480a-46ac-bfce-f12c3b0c3098"). InnerVolumeSpecName "kube-api-access-7cph7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:12.396402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.396361    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:12.396402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.396392    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-console-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:12.396402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.396401    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:12.396402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.396410    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab583534-480a-46ac-bfce-f12c3b0c3098-console-oauth-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:12.396703 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.396418    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab583534-480a-46ac-bfce-f12c3b0c3098-oauth-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:12.396703 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:12.396426    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cph7\" (UniqueName: \"kubernetes.io/projected/ab583534-480a-46ac-bfce-f12c3b0c3098-kube-api-access-7cph7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:13.056325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.056285    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab583534-480a-46ac-bfce-f12c3b0c3098" containerID="994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545" exitCode=0
Sep 09 20:13:13.056762 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.056336    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc987686-46gkn"
Sep 09 20:13:13.056762 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.056375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc987686-46gkn" event={"ID":"ab583534-480a-46ac-bfce-f12c3b0c3098","Type":"ContainerDied","Data":"994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545"}
Sep 09 20:13:13.056762 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.056433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc987686-46gkn" event={"ID":"ab583534-480a-46ac-bfce-f12c3b0c3098","Type":"ContainerDied","Data":"ca6dc718d87ed29488a9a8119f620be7b330e1e545eb18f84852b7c2db481871"}
Sep 09 20:13:13.056762 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.056459    2563 scope.go:117] "RemoveContainer" containerID="994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545"
Sep 09 20:13:13.071857 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.071834    2563 scope.go:117] "RemoveContainer" containerID="994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545"
Sep 09 20:13:13.072326 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:13.072306    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545\": container with ID starting with 994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545 not found: ID does not exist" containerID="994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545"
Sep 09 20:13:13.072372 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.072335    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545"} err="failed to get container status \"994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545\": rpc error: code = NotFound desc = could not find container \"994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545\": container with ID starting with 994b1192c74d708c8673b8e886a43c62ad7c4dc6d76dcad2b99c11c5934e1545 not found: ID does not exist"
Sep 09 20:13:13.080497 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.080476    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cc987686-46gkn"]
Sep 09 20:13:13.083982 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.083962    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54cc987686-46gkn"]
Sep 09 20:13:13.424204 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:13.424175    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab583534-480a-46ac-bfce-f12c3b0c3098" path="/var/lib/kubelet/pods/ab583534-480a-46ac-bfce-f12c3b0c3098/volumes"
Sep 09 20:13:35.166777 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:35.166745    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:13:35.167431 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:35.167034    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="alertmanager" containerID="cri-o://057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b" gracePeriod=120
Sep 09 20:13:35.167431 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:35.167120    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-web" containerID="cri-o://bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0" gracePeriod=120
Sep 09 20:13:35.167592 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:35.167568    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-metric" containerID="cri-o://09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b" gracePeriod=120
Sep 09 20:13:35.167760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:35.167734    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="config-reloader" containerID="cri-o://cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93" gracePeriod=120
Sep 09 20:13:35.167874 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:35.167748    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="prom-label-proxy" containerID="cri-o://4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a" gracePeriod=120
Sep 09 20:13:35.167874 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:35.167859    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy" containerID="cri-o://78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d" gracePeriod=120
Sep 09 20:13:36.142741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142709    2563 generic.go:356] "Generic (PLEG): container finished" podID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerID="4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a" exitCode=0
Sep 09 20:13:36.142741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142736    2563 generic.go:356] "Generic (PLEG): container finished" podID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerID="09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b" exitCode=0
Sep 09 20:13:36.142741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142742    2563 generic.go:356] "Generic (PLEG): container finished" podID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerID="78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d" exitCode=0
Sep 09 20:13:36.142741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142748    2563 generic.go:356] "Generic (PLEG): container finished" podID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerID="cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93" exitCode=0
Sep 09 20:13:36.142741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142753    2563 generic.go:356] "Generic (PLEG): container finished" podID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerID="057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b" exitCode=0
Sep 09 20:13:36.143109 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a"}
Sep 09 20:13:36.143109 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b"}
Sep 09 20:13:36.143109 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d"}
Sep 09 20:13:36.143109 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93"}
Sep 09 20:13:36.143109 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.142988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b"}
Sep 09 20:13:36.427733 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.427642    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:36.597170 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-main-db\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597170 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597199    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-out\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-volume\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-web\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-metrics-client-ca\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-web-config\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvk5q\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-kube-api-access-wvk5q\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:13:36.597642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597544    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-tls-assets\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-cluster-tls-config\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597981 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle\") pod \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\" (UID: \"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400\") "
Sep 09 20:13:36.597981 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:36.598101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.597980    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-main-db\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.598101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.598100    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-metrics-client-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.599325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.599114    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:36.600353 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.600325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:36.600785 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.600756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:36.601112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.601073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-out" (OuterVolumeSpecName: "config-out") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:13:36.601206 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.601173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-kube-api-access-wvk5q" (OuterVolumeSpecName: "kube-api-access-wvk5q") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "kube-api-access-wvk5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:36.601788 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.601762    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:36.602001 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.601979    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-volume" (OuterVolumeSpecName: "config-volume") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:36.602159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.602143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:36.602360 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.602343    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:36.605933 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.605905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:36.612424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.612401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-web-config" (OuterVolumeSpecName: "web-config") pod "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" (UID: "ccfe2555-e18f-4bc0-b5c0-8a71e23f5400"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698849    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698888    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-out\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698898    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698908    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698919    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-main-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698927    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-web-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698935    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvk5q\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-kube-api-access-wvk5q\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698943    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698951    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-tls-assets\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698959    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-cluster-tls-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:36.698943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:36.698976    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:13:37.149873 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.149841    2563 generic.go:356] "Generic (PLEG): container finished" podID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerID="bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0" exitCode=0
Sep 09 20:13:37.150036 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.149913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0"}
Sep 09 20:13:37.150036 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.149919    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.150036 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.150007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ccfe2555-e18f-4bc0-b5c0-8a71e23f5400","Type":"ContainerDied","Data":"77d2fc75d2956c7dd8266167b3d1b14cd11ab7ce8a5ed9e0057ee027c8846186"}
Sep 09 20:13:37.150155 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.150041    2563 scope.go:117] "RemoveContainer" containerID="4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a"
Sep 09 20:13:37.159302 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.159284    2563 scope.go:117] "RemoveContainer" containerID="09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b"
Sep 09 20:13:37.166263 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.166244    2563 scope.go:117] "RemoveContainer" containerID="78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d"
Sep 09 20:13:37.172728 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.172699    2563 scope.go:117] "RemoveContainer" containerID="bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0"
Sep 09 20:13:37.174420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.174400    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:13:37.179544 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.179526    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:13:37.179861 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.179846    2563 scope.go:117] "RemoveContainer" containerID="cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93"
Sep 09 20:13:37.186598 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.186581    2563 scope.go:117] "RemoveContainer" containerID="057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b"
Sep 09 20:13:37.193342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.193327    2563 scope.go:117] "RemoveContainer" containerID="66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32"
Sep 09 20:13:37.199903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.199883    2563 scope.go:117] "RemoveContainer" containerID="4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a"
Sep 09 20:13:37.200153 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:37.200134    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a\": container with ID starting with 4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a not found: ID does not exist" containerID="4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a"
Sep 09 20:13:37.200198 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.200161    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a"} err="failed to get container status \"4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a\": rpc error: code = NotFound desc = could not find container \"4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a\": container with ID starting with 4bfa67e169b58514ed78fc8276fdd530aca29d11fd70b68b707e1c32e2df4c7a not found: ID does not exist"
Sep 09 20:13:37.200198 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.200179    2563 scope.go:117] "RemoveContainer" containerID="09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b"
Sep 09 20:13:37.200452 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:37.200432    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b\": container with ID starting with 09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b not found: ID does not exist" containerID="09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b"
Sep 09 20:13:37.200502 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.200460    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b"} err="failed to get container status \"09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b\": rpc error: code = NotFound desc = could not find container \"09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b\": container with ID starting with 09436c332497b8da0d7a07779a5c5bd6753fe0073a9f32d22fbc04601ca9634b not found: ID does not exist"
Sep 09 20:13:37.200502 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.200477    2563 scope.go:117] "RemoveContainer" containerID="78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d"
Sep 09 20:13:37.200767 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:37.200751    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d\": container with ID starting with 78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d not found: ID does not exist" containerID="78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d"
Sep 09 20:13:37.200812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.200772    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d"} err="failed to get container status \"78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d\": rpc error: code = NotFound desc = could not find container \"78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d\": container with ID starting with 78b1910cd7b62e1ce3d346dc96c4049a8af74899592285688aeeb9e1e4c9866d not found: ID does not exist"
Sep 09 20:13:37.200812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.200786    2563 scope.go:117] "RemoveContainer" containerID="bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0"
Sep 09 20:13:37.201053 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:37.201021    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0\": container with ID starting with bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0 not found: ID does not exist" containerID="bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0"
Sep 09 20:13:37.201112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.201045    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0"} err="failed to get container status \"bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0\": rpc error: code = NotFound desc = could not find container \"bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0\": container with ID starting with bf33456285b02bfcf94d21374af0042388a1b4f077a019dd9242a5a577c18bf0 not found: ID does not exist"
Sep 09 20:13:37.201112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.201066    2563 scope.go:117] "RemoveContainer" containerID="cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93"
Sep 09 20:13:37.201414 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:37.201351    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93\": container with ID starting with cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93 not found: ID does not exist" containerID="cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93"
Sep 09 20:13:37.201414 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.201390    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93"} err="failed to get container status \"cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93\": rpc error: code = NotFound desc = could not find container \"cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93\": container with ID starting with cd517472033de9d0a6ee5c5f8f6701708b2dd42138999d46b7bf5b2b57c0da93 not found: ID does not exist"
Sep 09 20:13:37.201414 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.201411    2563 scope.go:117] "RemoveContainer" containerID="057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b"
Sep 09 20:13:37.201803 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:37.201779    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b\": container with ID starting with 057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b not found: ID does not exist" containerID="057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b"
Sep 09 20:13:37.201894 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.201809    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b"} err="failed to get container status \"057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b\": rpc error: code = NotFound desc = could not find container \"057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b\": container with ID starting with 057904ddd7103787012696d059908ef48b7bcf1634b96f40160313bc0fb1a55b not found: ID does not exist"
Sep 09 20:13:37.201894 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.201829    2563 scope.go:117] "RemoveContainer" containerID="66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32"
Sep 09 20:13:37.202141 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:13:37.202122    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32\": container with ID starting with 66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32 not found: ID does not exist" containerID="66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32"
Sep 09 20:13:37.202208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.202143    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32"} err="failed to get container status \"66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32\": rpc error: code = NotFound desc = could not find container \"66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32\": container with ID starting with 66baf1c748d991ee970eede05fc63c7b92c1c81662499502ac4ae3db2ede1e32 not found: ID does not exist"
Sep 09 20:13:37.203183 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203168    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:13:37.203416 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203405    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="prom-label-proxy"
Sep 09 20:13:37.203450 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203419    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="prom-label-proxy"
Sep 09 20:13:37.203450 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203426    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="alertmanager"
Sep 09 20:13:37.203450 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203432    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="alertmanager"
Sep 09 20:13:37.203450 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-metric"
Sep 09 20:13:37.203450 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-metric"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203452    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab583534-480a-46ac-bfce-f12c3b0c3098" containerName="console"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203458    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab583534-480a-46ac-bfce-f12c3b0c3098" containerName="console"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203464    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="init-config-reloader"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203470    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="init-config-reloader"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203477    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203482    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203495    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="config-reloader"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="config-reloader"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203508    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-web"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203513    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-web"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203553    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-metric"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203561    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy-web"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203566    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab583534-480a-46ac-bfce-f12c3b0c3098" containerName="console"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203572    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="alertmanager"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203577    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="kube-rbac-proxy"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203582    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="prom-label-proxy"
Sep 09 20:13:37.203623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.203587    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" containerName="config-reloader"
Sep 09 20:13:37.206597 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.206583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.209281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209251    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:13:37.209391 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209257    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:13:37.209391 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:13:37.209391 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209258    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:13:37.209576 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:13:37.209576 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209533    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:13:37.209837 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209814    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:13:37.209931 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209855    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9frg8\""
Sep 09 20:13:37.209931 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.209821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:13:37.215239 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.215222    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:13:37.220212 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.220189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:13:37.304098 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf9a7ce-6192-4305-8822-b441a0427804-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304098 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9a7ce-6192-4305-8822-b441a0427804-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304306 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304306 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf9a7ce-6192-4305-8822-b441a0427804-config-out\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304306 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304306 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-web-config\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf9a7ce-6192-4305-8822-b441a0427804-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9xz\" (UniqueName: \"kubernetes.io/projected/ddf9a7ce-6192-4305-8822-b441a0427804-kube-api-access-2c9xz\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf9a7ce-6192-4305-8822-b441a0427804-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.304623 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.304543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf9a7ce-6192-4305-8822-b441a0427804-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9a7ce-6192-4305-8822-b441a0427804-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf9a7ce-6192-4305-8822-b441a0427804-config-out\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-web-config\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf9a7ce-6192-4305-8822-b441a0427804-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xz\" (UniqueName: \"kubernetes.io/projected/ddf9a7ce-6192-4305-8822-b441a0427804-kube-api-access-2c9xz\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf9a7ce-6192-4305-8822-b441a0427804-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.405834 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.405808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.406754 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.406596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9a7ce-6192-4305-8822-b441a0427804-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.406884 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.406860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf9a7ce-6192-4305-8822-b441a0427804-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.408983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.408684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf9a7ce-6192-4305-8822-b441a0427804-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.408983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.408714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf9a7ce-6192-4305-8822-b441a0427804-config-out\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.408983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.408787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.408983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.408947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf9a7ce-6192-4305-8822-b441a0427804-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.409237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.409221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.409526 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.409503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.409880 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.409857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-web-config\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.410060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.410042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.410121 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.410107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.410818 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.410799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddf9a7ce-6192-4305-8822-b441a0427804-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.413524 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.413503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9xz\" (UniqueName: \"kubernetes.io/projected/ddf9a7ce-6192-4305-8822-b441a0427804-kube-api-access-2c9xz\") pod \"alertmanager-main-0\" (UID: \"ddf9a7ce-6192-4305-8822-b441a0427804\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.424341 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.424315    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccfe2555-e18f-4bc0-b5c0-8a71e23f5400" path="/var/lib/kubelet/pods/ccfe2555-e18f-4bc0-b5c0-8a71e23f5400/volumes"
Sep 09 20:13:37.515999 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.515966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:13:37.791855 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:37.791736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:13:37.794466 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:13:37.794438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf9a7ce_6192_4305_8822_b441a0427804.slice/crio-5e38f73d49b0bc1d4895eb5911d7e835e7c21fa08986c673562e3165a09c3791 WatchSource:0}: Error finding container 5e38f73d49b0bc1d4895eb5911d7e835e7c21fa08986c673562e3165a09c3791: Status 404 returned error can't find the container with id 5e38f73d49b0bc1d4895eb5911d7e835e7c21fa08986c673562e3165a09c3791
Sep 09 20:13:38.155666 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:38.155618    2563 generic.go:356] "Generic (PLEG): container finished" podID="ddf9a7ce-6192-4305-8822-b441a0427804" containerID="c752d1357b692d48e1f550658273e54805c575f3e48449a23f30ba5b4997dc26" exitCode=0
Sep 09 20:13:38.155848 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:38.155672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerDied","Data":"c752d1357b692d48e1f550658273e54805c575f3e48449a23f30ba5b4997dc26"}
Sep 09 20:13:38.155848 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:38.155766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerStarted","Data":"5e38f73d49b0bc1d4895eb5911d7e835e7c21fa08986c673562e3165a09c3791"}
Sep 09 20:13:39.162078 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:39.162045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerStarted","Data":"f08ef1d1da6b2d5c4028a224cb23978c366f242b16e217b411798079107900f3"}
Sep 09 20:13:39.162455 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:39.162109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerStarted","Data":"1d34fc9960e3f79b4b47b75205e7718c34b632d6c7e16a20aeb32c5800eb8c47"}
Sep 09 20:13:39.162455 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:39.162134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerStarted","Data":"3e0f1d9e8e08c5f3bcfd4ff087e5c382e9242d16e71ffc5e41e4e605aad9f817"}
Sep 09 20:13:39.162455 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:39.162164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerStarted","Data":"8976d98ef0aa93a353ea5191d63ffd17f530cd22f785b1245e7ece768ff15e73"}
Sep 09 20:13:39.162455 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:39.162190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerStarted","Data":"b8998c125d6ddcbda8584f962b15fce22fe83d401bb46ab825c2c750aacf5023"}
Sep 09 20:13:39.162455 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:39.162213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddf9a7ce-6192-4305-8822-b441a0427804","Type":"ContainerStarted","Data":"a80c9de9f19e319f4f081504d5e4d105734cd1f2c23d86ba8429184af2bc24f3"}
Sep 09 20:13:39.194846 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:13:39.194800    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.179950598 podStartE2EDuration="2.19478437s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.833772ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:39.194433345 +0000 UTC m=+146.233603333" watchObservedRunningTime="2026-09-09 20:13:39.19478437 +0000 UTC m=+146.233954321"
Sep 09 20:14:14.372375 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.372341    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8874d6dbd-wgtch"]
Sep 09 20:14:14.375204 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.375187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.393416 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.393365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8874d6dbd-wgtch"]
Sep 09 20:14:14.478400 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.478371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-oauth-serving-cert\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.478578 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.478424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68x6\" (UniqueName: \"kubernetes.io/projected/226e2d19-bbfe-4673-98a5-68ef2d9a7721-kube-api-access-z68x6\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.478578 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.478459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-trusted-ca-bundle\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.478578 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.478485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-service-ca\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.478578 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.478545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-oauth-config\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.478776 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.478585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-serving-cert\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.478776 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.478626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-config\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.579186 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.579148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-serving-cert\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.579186 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.579189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-config\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.579431 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.579211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-oauth-serving-cert\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.579431 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.579251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68x6\" (UniqueName: \"kubernetes.io/projected/226e2d19-bbfe-4673-98a5-68ef2d9a7721-kube-api-access-z68x6\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.579431 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.579410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-trusted-ca-bundle\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.579618 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.579481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-service-ca\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.579618 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.579519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-oauth-config\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.580057 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.580032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-config\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.580388 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.580052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-oauth-serving-cert\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.580488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.580088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-service-ca\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.580488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.580408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-trusted-ca-bundle\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.581864 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.581844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-serving-cert\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.582014 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.581997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-oauth-config\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.648003 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.647937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68x6\" (UniqueName: \"kubernetes.io/projected/226e2d19-bbfe-4673-98a5-68ef2d9a7721-kube-api-access-z68x6\") pod \"console-8874d6dbd-wgtch\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") " pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.684266 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.684234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:14.954207 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:14.954133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8874d6dbd-wgtch"]
Sep 09 20:14:14.956894 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:14:14.956869    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226e2d19_bbfe_4673_98a5_68ef2d9a7721.slice/crio-483cb7d32238b89fbb1415e10fe9c370af6b1eefd684619f6125b6a9ccc84da9 WatchSource:0}: Error finding container 483cb7d32238b89fbb1415e10fe9c370af6b1eefd684619f6125b6a9ccc84da9: Status 404 returned error can't find the container with id 483cb7d32238b89fbb1415e10fe9c370af6b1eefd684619f6125b6a9ccc84da9
Sep 09 20:14:15.292799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:15.292710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8874d6dbd-wgtch" event={"ID":"226e2d19-bbfe-4673-98a5-68ef2d9a7721","Type":"ContainerStarted","Data":"76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae"}
Sep 09 20:14:15.292799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:15.292762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8874d6dbd-wgtch" event={"ID":"226e2d19-bbfe-4673-98a5-68ef2d9a7721","Type":"ContainerStarted","Data":"483cb7d32238b89fbb1415e10fe9c370af6b1eefd684619f6125b6a9ccc84da9"}
Sep 09 20:14:15.315851 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:15.315801    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8874d6dbd-wgtch" podStartSLOduration=1.3157847999999999 podStartE2EDuration="1.3157848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:15.314502677 +0000 UTC m=+182.353672638" watchObservedRunningTime="2026-09-09 20:14:15.3157848 +0000 UTC m=+182.354954750"
Sep 09 20:14:24.684733 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:24.684689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:24.685253 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:24.684784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:24.689433 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:24.689410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:25.333027 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:25.333001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:14:25.376923 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:25.376895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f99ff8bb-x7gst"]
Sep 09 20:14:41.933096 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:41.933060    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"]
Sep 09 20:14:41.935608 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:41.935591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:41.938435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:41.938416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:14:41.938435 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:41.938425    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:14:41.939683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:41.939634    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-44dgn\""
Sep 09 20:14:41.944978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:41.944954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"]
Sep 09 20:14:42.018394 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.018356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.018394 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.018393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.018633 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.018419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcg96\" (UniqueName: \"kubernetes.io/projected/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-kube-api-access-xcg96\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.118882 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.118842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.118882 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.118882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.119105 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.118913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcg96\" (UniqueName: \"kubernetes.io/projected/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-kube-api-access-xcg96\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.119266 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.119246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.119266 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.119261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.128008 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.127972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcg96\" (UniqueName: \"kubernetes.io/projected/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-kube-api-access-xcg96\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.245229 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.245136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:14:42.514991 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:42.514965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"]
Sep 09 20:14:42.517429 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:14:42.517396    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd946d0ee_e6bc_4cbf_b5f4_f48b3d11ac42.slice/crio-9b8b29e4ac57347b78cf5e39d2272bbcaba8572b9bf77c14232a1e09409cf3f1 WatchSource:0}: Error finding container 9b8b29e4ac57347b78cf5e39d2272bbcaba8572b9bf77c14232a1e09409cf3f1: Status 404 returned error can't find the container with id 9b8b29e4ac57347b78cf5e39d2272bbcaba8572b9bf77c14232a1e09409cf3f1
Sep 09 20:14:43.393896 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:43.393860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn" event={"ID":"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42","Type":"ContainerStarted","Data":"9b8b29e4ac57347b78cf5e39d2272bbcaba8572b9bf77c14232a1e09409cf3f1"}
Sep 09 20:14:50.399060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.398988    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f99ff8bb-x7gst" podUID="3eb19115-e909-4d75-8984-8bd247b2e8b7" containerName="console" containerID="cri-o://21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a" gracePeriod=15
Sep 09 20:14:50.427090 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.427056    2563 generic.go:356] "Generic (PLEG): container finished" podID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerID="ba7c76100e2ebbd4de0c774b5f171d3b1ca8ae2de52ed4fa267466c598013920" exitCode=0
Sep 09 20:14:50.427175 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.427137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn" event={"ID":"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42","Type":"ContainerDied","Data":"ba7c76100e2ebbd4de0c774b5f171d3b1ca8ae2de52ed4fa267466c598013920"}
Sep 09 20:14:50.637577 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.637551    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:14:50.684498 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkt7t\" (UniqueName: \"kubernetes.io/projected/3eb19115-e909-4d75-8984-8bd247b2e8b7-kube-api-access-pkt7t\") pod \"3eb19115-e909-4d75-8984-8bd247b2e8b7\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") "
Sep 09 20:14:50.684498 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-oauth-config\") pod \"3eb19115-e909-4d75-8984-8bd247b2e8b7\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") "
Sep 09 20:14:50.684498 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684489    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-oauth-serving-cert\") pod \"3eb19115-e909-4d75-8984-8bd247b2e8b7\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") "
Sep 09 20:14:50.684808 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-serving-cert\") pod \"3eb19115-e909-4d75-8984-8bd247b2e8b7\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") "
Sep 09 20:14:50.684808 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684630    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-trusted-ca-bundle\") pod \"3eb19115-e909-4d75-8984-8bd247b2e8b7\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") "
Sep 09 20:14:50.684808 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-config\") pod \"3eb19115-e909-4d75-8984-8bd247b2e8b7\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") "
Sep 09 20:14:50.684808 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-service-ca\") pod \"3eb19115-e909-4d75-8984-8bd247b2e8b7\" (UID: \"3eb19115-e909-4d75-8984-8bd247b2e8b7\") "
Sep 09 20:14:50.684999 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.684883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3eb19115-e909-4d75-8984-8bd247b2e8b7" (UID: "3eb19115-e909-4d75-8984-8bd247b2e8b7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.685104 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.685006    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3eb19115-e909-4d75-8984-8bd247b2e8b7" (UID: "3eb19115-e909-4d75-8984-8bd247b2e8b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.685192 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.685174    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-oauth-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:14:50.685242 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.685198    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:14:50.685242 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.685194    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-service-ca" (OuterVolumeSpecName: "service-ca") pod "3eb19115-e909-4d75-8984-8bd247b2e8b7" (UID: "3eb19115-e909-4d75-8984-8bd247b2e8b7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.685320 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.685184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-config" (OuterVolumeSpecName: "console-config") pod "3eb19115-e909-4d75-8984-8bd247b2e8b7" (UID: "3eb19115-e909-4d75-8984-8bd247b2e8b7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.686847 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.686826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3eb19115-e909-4d75-8984-8bd247b2e8b7" (UID: "3eb19115-e909-4d75-8984-8bd247b2e8b7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:50.686847 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.686836    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb19115-e909-4d75-8984-8bd247b2e8b7-kube-api-access-pkt7t" (OuterVolumeSpecName: "kube-api-access-pkt7t") pod "3eb19115-e909-4d75-8984-8bd247b2e8b7" (UID: "3eb19115-e909-4d75-8984-8bd247b2e8b7"). InnerVolumeSpecName "kube-api-access-pkt7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:50.686957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.686908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3eb19115-e909-4d75-8984-8bd247b2e8b7" (UID: "3eb19115-e909-4d75-8984-8bd247b2e8b7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:50.786413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.786389    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:14:50.786413 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.786413    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb19115-e909-4d75-8984-8bd247b2e8b7-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:14:50.786564 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.786422    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkt7t\" (UniqueName: \"kubernetes.io/projected/3eb19115-e909-4d75-8984-8bd247b2e8b7-kube-api-access-pkt7t\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:14:50.786564 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.786431    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-oauth-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:14:50.786564 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:50.786439    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb19115-e909-4d75-8984-8bd247b2e8b7-console-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.437993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.437956    2563 generic.go:356] "Generic (PLEG): container finished" podID="3eb19115-e909-4d75-8984-8bd247b2e8b7" containerID="21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a" exitCode=0
Sep 09 20:14:51.438374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.438035    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f99ff8bb-x7gst"
Sep 09 20:14:51.438374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.438056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f99ff8bb-x7gst" event={"ID":"3eb19115-e909-4d75-8984-8bd247b2e8b7","Type":"ContainerDied","Data":"21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a"}
Sep 09 20:14:51.438374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.438138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f99ff8bb-x7gst" event={"ID":"3eb19115-e909-4d75-8984-8bd247b2e8b7","Type":"ContainerDied","Data":"5f71fb4d59e4b6e65cef3bde745c970b170254649119014df283c4df66db7d35"}
Sep 09 20:14:51.438374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.438164    2563 scope.go:117] "RemoveContainer" containerID="21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a"
Sep 09 20:14:51.447837 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.447815    2563 scope.go:117] "RemoveContainer" containerID="21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a"
Sep 09 20:14:51.448190 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:14:51.448166    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a\": container with ID starting with 21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a not found: ID does not exist" containerID="21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a"
Sep 09 20:14:51.448291 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.448201    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a"} err="failed to get container status \"21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a\": rpc error: code = NotFound desc = could not find container \"21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a\": container with ID starting with 21c85ae30ab78d3f5620e1b31a6ace7a1e0f08bb52150e7ca55e33ddcbc1959a not found: ID does not exist"
Sep 09 20:14:51.456383 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.456361    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f99ff8bb-x7gst"]
Sep 09 20:14:51.459574 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:51.459551    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f99ff8bb-x7gst"]
Sep 09 20:14:53.423596 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:53.423554    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb19115-e909-4d75-8984-8bd247b2e8b7" path="/var/lib/kubelet/pods/3eb19115-e909-4d75-8984-8bd247b2e8b7/volumes"
Sep 09 20:14:53.447352 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:53.447320    2563 generic.go:356] "Generic (PLEG): container finished" podID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerID="60820823c3753d1a12f9ade0f5b7e4f7c1884b24f1d0dc3d13a23c853d877575" exitCode=0
Sep 09 20:14:53.447520 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:14:53.447400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn" event={"ID":"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42","Type":"ContainerDied","Data":"60820823c3753d1a12f9ade0f5b7e4f7c1884b24f1d0dc3d13a23c853d877575"}
Sep 09 20:15:01.485510 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:01.485474    2563 generic.go:356] "Generic (PLEG): container finished" podID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerID="1b01341719497ec64ac7a94170225481850aa0138e4b5bbd18c7bb08215c74f1" exitCode=0
Sep 09 20:15:01.485951 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:01.485526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn" event={"ID":"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42","Type":"ContainerDied","Data":"1b01341719497ec64ac7a94170225481850aa0138e4b5bbd18c7bb08215c74f1"}
Sep 09 20:15:02.612838 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.612816    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:15:02.684363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.684332    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcg96\" (UniqueName: \"kubernetes.io/projected/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-kube-api-access-xcg96\") pod \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") "
Sep 09 20:15:02.684522 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.684390    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-util\") pod \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") "
Sep 09 20:15:02.684522 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.684445    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-bundle\") pod \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\" (UID: \"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42\") "
Sep 09 20:15:02.685116 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.685086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-bundle" (OuterVolumeSpecName: "bundle") pod "d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" (UID: "d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:02.686777 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.686736    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-kube-api-access-xcg96" (OuterVolumeSpecName: "kube-api-access-xcg96") pod "d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" (UID: "d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42"). InnerVolumeSpecName "kube-api-access-xcg96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:15:02.688991 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.688971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-util" (OuterVolumeSpecName: "util") pod "d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" (UID: "d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:02.785484 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.785381    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-util\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:15:02.785484 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.785424    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:15:02.785484 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:02.785434    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcg96\" (UniqueName: \"kubernetes.io/projected/d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42-kube-api-access-xcg96\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:15:03.494963 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:03.494930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn" event={"ID":"d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42","Type":"ContainerDied","Data":"9b8b29e4ac57347b78cf5e39d2272bbcaba8572b9bf77c14232a1e09409cf3f1"}
Sep 09 20:15:03.494963 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:03.494953    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag22gn"
Sep 09 20:15:03.495154 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:03.494980    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8b29e4ac57347b78cf5e39d2272bbcaba8572b9bf77c14232a1e09409cf3f1"
Sep 09 20:15:09.501178 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501138    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"]
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501536    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerName="pull"
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501553    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerName="pull"
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerName="util"
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501583    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerName="util"
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb19115-e909-4d75-8984-8bd247b2e8b7" containerName="console"
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501602    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb19115-e909-4d75-8984-8bd247b2e8b7" containerName="console"
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501612    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerName="extract"
Sep 09 20:15:09.501642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501619    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerName="extract"
Sep 09 20:15:09.502066 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501710    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb19115-e909-4d75-8984-8bd247b2e8b7" containerName="console"
Sep 09 20:15:09.502066 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.501725    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d946d0ee-e6bc-4cbf-b5f4-f48b3d11ac42" containerName="extract"
Sep 09 20:15:09.549007 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.548973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"]
Sep 09 20:15:09.549164 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.549084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:09.552415 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.552394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:15:09.552522 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.552503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:15:09.553400 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.553383    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-v7fjk\""
Sep 09 20:15:09.553499 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.553405    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:15:09.635808 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.635781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7587fe80-c65a-4e9d-a2c0-07bac4dfe73f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-746j8\" (UID: \"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:09.635963 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.635824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xgdp\" (UniqueName: \"kubernetes.io/projected/7587fe80-c65a-4e9d-a2c0-07bac4dfe73f-kube-api-access-8xgdp\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-746j8\" (UID: \"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:09.736332 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.736298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7587fe80-c65a-4e9d-a2c0-07bac4dfe73f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-746j8\" (UID: \"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:09.736494 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.736343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xgdp\" (UniqueName: \"kubernetes.io/projected/7587fe80-c65a-4e9d-a2c0-07bac4dfe73f-kube-api-access-8xgdp\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-746j8\" (UID: \"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:09.738762 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.738734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7587fe80-c65a-4e9d-a2c0-07bac4dfe73f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-746j8\" (UID: \"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:09.744999 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.744968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xgdp\" (UniqueName: \"kubernetes.io/projected/7587fe80-c65a-4e9d-a2c0-07bac4dfe73f-kube-api-access-8xgdp\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-746j8\" (UID: \"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:09.859860 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:09.859823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:10.148824 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:10.148769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"]
Sep 09 20:15:10.151659 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:15:10.151619    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7587fe80_c65a_4e9d_a2c0_07bac4dfe73f.slice/crio-7bd3c6ee72a6328c6f8dab23bc11b90f00514204431a720c1174ac43cf644101 WatchSource:0}: Error finding container 7bd3c6ee72a6328c6f8dab23bc11b90f00514204431a720c1174ac43cf644101: Status 404 returned error can't find the container with id 7bd3c6ee72a6328c6f8dab23bc11b90f00514204431a720c1174ac43cf644101
Sep 09 20:15:10.522001 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:10.521913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8" event={"ID":"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f","Type":"ContainerStarted","Data":"7bd3c6ee72a6328c6f8dab23bc11b90f00514204431a720c1174ac43cf644101"}
Sep 09 20:15:13.537115 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.537081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8" event={"ID":"7587fe80-c65a-4e9d-a2c0-07bac4dfe73f","Type":"ContainerStarted","Data":"1be16afc141f8da5557432a0453b7d968d3152c87cefb1dc6ba8041f924a5b3c"}
Sep 09 20:15:13.537504 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.537155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:13.556984 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.556927    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8" podStartSLOduration=1.736353855 podStartE2EDuration="4.556906107s" totalImagesPullingTime="2.820552252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:13.554916155 +0000 UTC m=+240.594086105" watchObservedRunningTime="2026-09-09 20:15:13.556906107 +0000 UTC m=+240.596076059"
Sep 09 20:15:13.733471 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.733394    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c9mkl"]
Sep 09 20:15:13.736377 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.736357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.739063 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.739041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:15:13.739176 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.739073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-qkmzm\""
Sep 09 20:15:13.739176 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.739048    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:15:13.745327 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.745303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c9mkl"]
Sep 09 20:15:13.874921 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.874882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-cabundle0\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.875086 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.874940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558r7\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-kube-api-access-558r7\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.875086 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.874974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.939704 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.939673    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"]
Sep 09 20:15:13.941845 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.941831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:13.944628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.944610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:15:13.953033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.953010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"]
Sep 09 20:15:13.975909 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.975877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-cabundle0\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.976043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.975917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558r7\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-kube-api-access-558r7\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.976043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.975937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.976043 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:13.976031    2563 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 09 20:15:13.976043 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:13.976044    2563 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:15:13.976171 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:13.976052    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:15:13.976171 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:13.976063    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-c9mkl: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 09 20:15:13.976171 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:13.976111    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates podName:5aa4ae2c-59dc-4afa-9234-1afc2857ecd7 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:14.476097594 +0000 UTC m=+241.515267522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates") pod "keda-operator-579c8958f7-c9mkl" (UID: "5aa4ae2c-59dc-4afa-9234-1afc2857ecd7") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 09 20:15:13.976543 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.976523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-cabundle0\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:13.987237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:13.987173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-558r7\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-kube-api-access-558r7\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:14.077104 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.077068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.077104 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.077101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p745p\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-kube-api-access-p745p\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.077298 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.077223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e74f9282-9069-4aa8-a2f5-17846cb0d66f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.178043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.177999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.178043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.178044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p745p\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-kube-api-access-p745p\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.178300 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.178139    2563 secret.go:281] references non-existent secret key: tls.crt
Sep 09 20:15:14.178300 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.178159    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 09 20:15:14.178300 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.178178    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd: references non-existent secret key: tls.crt
Sep 09 20:15:14.178300 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.178197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e74f9282-9069-4aa8-a2f5-17846cb0d66f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.178300 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.178231    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates podName:e74f9282-9069-4aa8-a2f5-17846cb0d66f nodeName:}" failed. No retries permitted until 2026-09-09 20:15:14.678215659 +0000 UTC m=+241.717385588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates") pod "keda-metrics-apiserver-77857b7b98-z8vzd" (UID: "e74f9282-9069-4aa8-a2f5-17846cb0d66f") : references non-existent secret key: tls.crt
Sep 09 20:15:14.178518 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.178495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e74f9282-9069-4aa8-a2f5-17846cb0d66f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.188799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.188771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p745p\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-kube-api-access-p745p\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.480344 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.480315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:14.480508 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.480490    2563 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:15:14.480549 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.480514    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:15:14.480549 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.480527    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-c9mkl: references non-existent secret key: ca.crt
Sep 09 20:15:14.480615 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.480595    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates podName:5aa4ae2c-59dc-4afa-9234-1afc2857ecd7 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:15.480575881 +0000 UTC m=+242.519745834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates") pod "keda-operator-579c8958f7-c9mkl" (UID: "5aa4ae2c-59dc-4afa-9234-1afc2857ecd7") : references non-existent secret key: ca.crt
Sep 09 20:15:14.681477 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.681445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:14.681899 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.681596    2563 secret.go:281] references non-existent secret key: tls.crt
Sep 09 20:15:14.681899 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.681616    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 09 20:15:14.681899 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.681632    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd: references non-existent secret key: tls.crt
Sep 09 20:15:14.681899 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:15:14.681709    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates podName:e74f9282-9069-4aa8-a2f5-17846cb0d66f nodeName:}" failed. No retries permitted until 2026-09-09 20:15:15.681692918 +0000 UTC m=+242.720862852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates") pod "keda-metrics-apiserver-77857b7b98-z8vzd" (UID: "e74f9282-9069-4aa8-a2f5-17846cb0d66f") : references non-existent secret key: tls.crt
Sep 09 20:15:14.711709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.711674    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-8nvfg"]
Sep 09 20:15:14.714080 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.714065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:14.716757 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.716723    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:15:14.723015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.722993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-8nvfg"]
Sep 09 20:15:14.783082 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.782992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vmn\" (UniqueName: \"kubernetes.io/projected/6ce48f61-25c9-486c-8d6c-e9094c334075-kube-api-access-h6vmn\") pod \"keda-admission-6c4d864b86-8nvfg\" (UID: \"6ce48f61-25c9-486c-8d6c-e9094c334075\") " pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:14.783256 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.783180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce48f61-25c9-486c-8d6c-e9094c334075-certificates\") pod \"keda-admission-6c4d864b86-8nvfg\" (UID: \"6ce48f61-25c9-486c-8d6c-e9094c334075\") " pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:14.884035 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.883996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce48f61-25c9-486c-8d6c-e9094c334075-certificates\") pod \"keda-admission-6c4d864b86-8nvfg\" (UID: \"6ce48f61-25c9-486c-8d6c-e9094c334075\") " pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:14.884237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.884048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vmn\" (UniqueName: \"kubernetes.io/projected/6ce48f61-25c9-486c-8d6c-e9094c334075-kube-api-access-h6vmn\") pod \"keda-admission-6c4d864b86-8nvfg\" (UID: \"6ce48f61-25c9-486c-8d6c-e9094c334075\") " pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:14.886818 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.886794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce48f61-25c9-486c-8d6c-e9094c334075-certificates\") pod \"keda-admission-6c4d864b86-8nvfg\" (UID: \"6ce48f61-25c9-486c-8d6c-e9094c334075\") " pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:14.892818 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:14.892791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vmn\" (UniqueName: \"kubernetes.io/projected/6ce48f61-25c9-486c-8d6c-e9094c334075-kube-api-access-h6vmn\") pod \"keda-admission-6c4d864b86-8nvfg\" (UID: \"6ce48f61-25c9-486c-8d6c-e9094c334075\") " pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:15.024625 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.024584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:15.306051 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.306013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-8nvfg"]
Sep 09 20:15:15.309940 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:15:15.309834    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce48f61_25c9_486c_8d6c_e9094c334075.slice/crio-0669e3178599eb67f552ecf5d630a3805f4d62c0e9a350410a4be08ca9698efd WatchSource:0}: Error finding container 0669e3178599eb67f552ecf5d630a3805f4d62c0e9a350410a4be08ca9698efd: Status 404 returned error can't find the container with id 0669e3178599eb67f552ecf5d630a3805f4d62c0e9a350410a4be08ca9698efd
Sep 09 20:15:15.488830 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.488799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:15.491405 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.491376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5aa4ae2c-59dc-4afa-9234-1afc2857ecd7-certificates\") pod \"keda-operator-579c8958f7-c9mkl\" (UID: \"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7\") " pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:15.546306 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.546282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:15.546479 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.546456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-8nvfg" event={"ID":"6ce48f61-25c9-486c-8d6c-e9094c334075","Type":"ContainerStarted","Data":"0669e3178599eb67f552ecf5d630a3805f4d62c0e9a350410a4be08ca9698efd"}
Sep 09 20:15:15.690533 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.690494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:15.693259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.693232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e74f9282-9069-4aa8-a2f5-17846cb0d66f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-z8vzd\" (UID: \"e74f9282-9069-4aa8-a2f5-17846cb0d66f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:15.753273 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.753234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:15.819368 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:15.819342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c9mkl"]
Sep 09 20:15:15.821747 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:15:15.821715    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa4ae2c_59dc_4afa_9234_1afc2857ecd7.slice/crio-adb5165a10314ffe386415e9047bb85c523db5855687875d1d69db359fff1aa2 WatchSource:0}: Error finding container adb5165a10314ffe386415e9047bb85c523db5855687875d1d69db359fff1aa2: Status 404 returned error can't find the container with id adb5165a10314ffe386415e9047bb85c523db5855687875d1d69db359fff1aa2
Sep 09 20:15:16.023712 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:16.023616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"]
Sep 09 20:15:16.033311 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:15:16.033061    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74f9282_9069_4aa8_a2f5_17846cb0d66f.slice/crio-ebfa43ccf6c48b65563e18383de039f305dc5c6389c52fa4cdb0d56d0f43e1bd WatchSource:0}: Error finding container ebfa43ccf6c48b65563e18383de039f305dc5c6389c52fa4cdb0d56d0f43e1bd: Status 404 returned error can't find the container with id ebfa43ccf6c48b65563e18383de039f305dc5c6389c52fa4cdb0d56d0f43e1bd
Sep 09 20:15:16.552675 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:16.552591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd" event={"ID":"e74f9282-9069-4aa8-a2f5-17846cb0d66f","Type":"ContainerStarted","Data":"ebfa43ccf6c48b65563e18383de039f305dc5c6389c52fa4cdb0d56d0f43e1bd"}
Sep 09 20:15:16.554091 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:16.554045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-c9mkl" event={"ID":"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7","Type":"ContainerStarted","Data":"adb5165a10314ffe386415e9047bb85c523db5855687875d1d69db359fff1aa2"}
Sep 09 20:15:17.561580 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:17.561490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-8nvfg" event={"ID":"6ce48f61-25c9-486c-8d6c-e9094c334075","Type":"ContainerStarted","Data":"54eecdb892963bdf4773c6d30a5733631491fae31cd1bedeaca829076138f759"}
Sep 09 20:15:17.562072 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:17.561726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:17.579240 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:17.579180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-8nvfg" podStartSLOduration=2.181121236 podStartE2EDuration="3.579159659s" totalImagesPullingTime="1.398038423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:17.578919206 +0000 UTC m=+244.618089163" watchObservedRunningTime="2026-09-09 20:15:17.579159659 +0000 UTC m=+244.618329608"
Sep 09 20:15:20.578028 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:20.577990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd" event={"ID":"e74f9282-9069-4aa8-a2f5-17846cb0d66f","Type":"ContainerStarted","Data":"21d3d29824cc50f09600c8dfd9c8419647a166a833cc4de390f21e21de0bec30"}
Sep 09 20:15:20.578473 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:20.578171    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:20.579846 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:20.579822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-c9mkl" event={"ID":"5aa4ae2c-59dc-4afa-9234-1afc2857ecd7","Type":"ContainerStarted","Data":"e351a3243860fbe6d839c30600068c910f9d87dc2e4b42bb95b37f8f8dc0d34b"}
Sep 09 20:15:20.579996 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:20.579962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:15:20.594306 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:20.594249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd" podStartSLOduration=3.635340758 podStartE2EDuration="7.594229405s" totalImagesPullingTime="3.958888647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:20.593978405 +0000 UTC m=+247.633148357" watchObservedRunningTime="2026-09-09 20:15:20.594229405 +0000 UTC m=+247.633399368"
Sep 09 20:15:20.610861 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:20.610810    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-c9mkl" podStartSLOduration=3.433174224 podStartE2EDuration="7.610794559s" totalImagesPullingTime="4.177620335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:20.609305223 +0000 UTC m=+247.648482999" watchObservedRunningTime="2026-09-09 20:15:20.610794559 +0000 UTC m=+247.649964510"
Sep 09 20:15:31.589749 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:31.589719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-z8vzd"
Sep 09 20:15:34.544313 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:34.544220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-746j8"
Sep 09 20:15:38.568852 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:38.568818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-8nvfg"
Sep 09 20:15:41.586940 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:15:41.586908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-c9mkl"
Sep 09 20:16:13.342349 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:13.342323    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:16:31.662394 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.662359    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-5mlvj"]
Sep 09 20:16:31.664461 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.664447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.667425 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.667401    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-2q6vh\""
Sep 09 20:16:31.667548 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.667443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:16:31.667548 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.667469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:16:31.668906 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.668889    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:16:31.670372 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.670354    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"]
Sep 09 20:16:31.672314 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.672294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:31.675019 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.674998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-vdjns\""
Sep 09 20:16:31.675117 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.675041    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:16:31.676349 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.676329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-5mlvj"]
Sep 09 20:16:31.685690 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.685668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"]
Sep 09 20:16:31.694108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.694088    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-q84gl"]
Sep 09 20:16:31.696247 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.696231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:31.699106 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.699085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:16:31.699287 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.699268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4rm66\""
Sep 09 20:16:31.705523 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.705502    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-q84gl"]
Sep 09 20:16:31.814754 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.814713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/390166ac-5157-4d7c-9463-b12ea47c3e23-data\") pod \"seaweedfs-64568bcd49-q84gl\" (UID: \"390166ac-5157-4d7c-9463-b12ea47c3e23\") " pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:31.814936 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.814765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce73316c-2447-4e6b-bf1c-fd5e00579cb4-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-skkp8\" (UID: \"ce73316c-2447-4e6b-bf1c-fd5e00579cb4\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:31.814936 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.814793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsgx\" (UniqueName: \"kubernetes.io/projected/390166ac-5157-4d7c-9463-b12ea47c3e23-kube-api-access-gmsgx\") pod \"seaweedfs-64568bcd49-q84gl\" (UID: \"390166ac-5157-4d7c-9463-b12ea47c3e23\") " pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:31.814936 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.814872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsl4g\" (UniqueName: \"kubernetes.io/projected/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-kube-api-access-dsl4g\") pod \"kserve-controller-manager-55b9566966-5mlvj\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") " pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.814936 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.814907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966vb\" (UniqueName: \"kubernetes.io/projected/ce73316c-2447-4e6b-bf1c-fd5e00579cb4-kube-api-access-966vb\") pod \"llmisvc-controller-manager-645f9c8fd6-skkp8\" (UID: \"ce73316c-2447-4e6b-bf1c-fd5e00579cb4\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:31.814936 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.814932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-cert\") pod \"kserve-controller-manager-55b9566966-5mlvj\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") " pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.915803 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.915716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966vb\" (UniqueName: \"kubernetes.io/projected/ce73316c-2447-4e6b-bf1c-fd5e00579cb4-kube-api-access-966vb\") pod \"llmisvc-controller-manager-645f9c8fd6-skkp8\" (UID: \"ce73316c-2447-4e6b-bf1c-fd5e00579cb4\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:31.915803 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.915757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-cert\") pod \"kserve-controller-manager-55b9566966-5mlvj\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") " pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.916010 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.915997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/390166ac-5157-4d7c-9463-b12ea47c3e23-data\") pod \"seaweedfs-64568bcd49-q84gl\" (UID: \"390166ac-5157-4d7c-9463-b12ea47c3e23\") " pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:31.916066 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.916022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce73316c-2447-4e6b-bf1c-fd5e00579cb4-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-skkp8\" (UID: \"ce73316c-2447-4e6b-bf1c-fd5e00579cb4\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:31.916066 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.916046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsgx\" (UniqueName: \"kubernetes.io/projected/390166ac-5157-4d7c-9463-b12ea47c3e23-kube-api-access-gmsgx\") pod \"seaweedfs-64568bcd49-q84gl\" (UID: \"390166ac-5157-4d7c-9463-b12ea47c3e23\") " pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:31.916157 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.916082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsl4g\" (UniqueName: \"kubernetes.io/projected/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-kube-api-access-dsl4g\") pod \"kserve-controller-manager-55b9566966-5mlvj\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") " pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.916466 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.916440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/390166ac-5157-4d7c-9463-b12ea47c3e23-data\") pod \"seaweedfs-64568bcd49-q84gl\" (UID: \"390166ac-5157-4d7c-9463-b12ea47c3e23\") " pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:31.918438 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.918413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-cert\") pod \"kserve-controller-manager-55b9566966-5mlvj\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") " pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.918584 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.918569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce73316c-2447-4e6b-bf1c-fd5e00579cb4-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-skkp8\" (UID: \"ce73316c-2447-4e6b-bf1c-fd5e00579cb4\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:31.924545 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.924495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966vb\" (UniqueName: \"kubernetes.io/projected/ce73316c-2447-4e6b-bf1c-fd5e00579cb4-kube-api-access-966vb\") pod \"llmisvc-controller-manager-645f9c8fd6-skkp8\" (UID: \"ce73316c-2447-4e6b-bf1c-fd5e00579cb4\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:31.925388 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.925369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsl4g\" (UniqueName: \"kubernetes.io/projected/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-kube-api-access-dsl4g\") pod \"kserve-controller-manager-55b9566966-5mlvj\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") " pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.925502 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.925485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsgx\" (UniqueName: \"kubernetes.io/projected/390166ac-5157-4d7c-9463-b12ea47c3e23-kube-api-access-gmsgx\") pod \"seaweedfs-64568bcd49-q84gl\" (UID: \"390166ac-5157-4d7c-9463-b12ea47c3e23\") " pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:31.974954 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.974925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:31.983957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:31.983932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:32.006146 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.006115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:32.250851 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.250773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-5mlvj"]
Sep 09 20:16:32.253818 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:16:32.253794    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c840d2_2f12_4a1a_93c2_fc0e94cea4dc.slice/crio-1ff8bd0519fbf9f9c150242bd026447c1d9ab0f9b18f463a21d72183dd04dca5 WatchSource:0}: Error finding container 1ff8bd0519fbf9f9c150242bd026447c1d9ab0f9b18f463a21d72183dd04dca5: Status 404 returned error can't find the container with id 1ff8bd0519fbf9f9c150242bd026447c1d9ab0f9b18f463a21d72183dd04dca5
Sep 09 20:16:32.255173 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.255157    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:16:32.265239 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.265221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"]
Sep 09 20:16:32.267220 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:16:32.267201    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce73316c_2447_4e6b_bf1c_fd5e00579cb4.slice/crio-61631b1467b459772d62fbf4bfcba0232a5be073c8df7415610ddabec3665214 WatchSource:0}: Error finding container 61631b1467b459772d62fbf4bfcba0232a5be073c8df7415610ddabec3665214: Status 404 returned error can't find the container with id 61631b1467b459772d62fbf4bfcba0232a5be073c8df7415610ddabec3665214
Sep 09 20:16:32.288436 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.288384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-q84gl"]
Sep 09 20:16:32.290793 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:16:32.290767    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390166ac_5157_4d7c_9463_b12ea47c3e23.slice/crio-1c05c0c95dffdeb961ab9cc281ccf7e675ff9194d4e95dd3e8493a4329d18ffe WatchSource:0}: Error finding container 1c05c0c95dffdeb961ab9cc281ccf7e675ff9194d4e95dd3e8493a4329d18ffe: Status 404 returned error can't find the container with id 1c05c0c95dffdeb961ab9cc281ccf7e675ff9194d4e95dd3e8493a4329d18ffe
Sep 09 20:16:32.886689 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.886632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-q84gl" event={"ID":"390166ac-5157-4d7c-9463-b12ea47c3e23","Type":"ContainerStarted","Data":"1c05c0c95dffdeb961ab9cc281ccf7e675ff9194d4e95dd3e8493a4329d18ffe"}
Sep 09 20:16:32.888089 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.888054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-5mlvj" event={"ID":"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc","Type":"ContainerStarted","Data":"1ff8bd0519fbf9f9c150242bd026447c1d9ab0f9b18f463a21d72183dd04dca5"}
Sep 09 20:16:32.889367 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:32.889302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8" event={"ID":"ce73316c-2447-4e6b-bf1c-fd5e00579cb4","Type":"ContainerStarted","Data":"61631b1467b459772d62fbf4bfcba0232a5be073c8df7415610ddabec3665214"}
Sep 09 20:16:37.918538 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.918503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-q84gl" event={"ID":"390166ac-5157-4d7c-9463-b12ea47c3e23","Type":"ContainerStarted","Data":"a474702127c4c9ecac0620bf563041ea6b9381486a1d90384f4e192221278b56"}
Sep 09 20:16:37.918992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.918687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:16:37.920249 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.920221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-5mlvj" event={"ID":"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc","Type":"ContainerStarted","Data":"5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a"}
Sep 09 20:16:37.920391 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.920288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:16:37.921804 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.921780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8" event={"ID":"ce73316c-2447-4e6b-bf1c-fd5e00579cb4","Type":"ContainerStarted","Data":"e917c03dfec90de4b81de7f72f28f02db83f626f873a95d05d07bdd21192a100"}
Sep 09 20:16:37.921905 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.921881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:16:37.934253 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.934211    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-q84gl" podStartSLOduration=2.24108439 podStartE2EDuration="6.934199131s" totalImagesPullingTime="4.693114741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:37.933774284 +0000 UTC m=+324.972944235" watchObservedRunningTime="2026-09-09 20:16:37.934199131 +0000 UTC m=+324.973369082"
Sep 09 20:16:37.948741 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.948700    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-5mlvj" podStartSLOduration=2.311078811 podStartE2EDuration="6.948687592s" totalImagesPullingTime="4.637608781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:37.947912429 +0000 UTC m=+324.987082381" watchObservedRunningTime="2026-09-09 20:16:37.948687592 +0000 UTC m=+324.987857542"
Sep 09 20:16:37.966692 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:37.966636    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8" podStartSLOduration=2.342304868 podStartE2EDuration="6.966618713s" totalImagesPullingTime="4.624313845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:37.965242596 +0000 UTC m=+325.004412546" watchObservedRunningTime="2026-09-09 20:16:37.966618713 +0000 UTC m=+325.005788664"
Sep 09 20:16:43.928707 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:16:43.928675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-q84gl"
Sep 09 20:17:08.929033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:08.928946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-skkp8"
Sep 09 20:17:08.932127 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:08.932098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:17:16.139764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.139727    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-5mlvj"]
Sep 09 20:17:16.140182 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.139938    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-55b9566966-5mlvj" podUID="a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" containerName="manager" containerID="cri-o://5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a" gracePeriod=10
Sep 09 20:17:16.161022 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.160996    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-s8fq2"]
Sep 09 20:17:16.163308 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.163293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.173711 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.173689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-s8fq2"]
Sep 09 20:17:16.270703 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.270673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b-cert\") pod \"kserve-controller-manager-55b9566966-s8fq2\" (UID: \"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b\") " pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.270859 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.270792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sf8b\" (UniqueName: \"kubernetes.io/projected/6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b-kube-api-access-5sf8b\") pod \"kserve-controller-manager-55b9566966-s8fq2\" (UID: \"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b\") " pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.371811 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.371777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sf8b\" (UniqueName: \"kubernetes.io/projected/6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b-kube-api-access-5sf8b\") pod \"kserve-controller-manager-55b9566966-s8fq2\" (UID: \"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b\") " pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.371905 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.371837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b-cert\") pod \"kserve-controller-manager-55b9566966-s8fq2\" (UID: \"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b\") " pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.374106 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.374084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b-cert\") pod \"kserve-controller-manager-55b9566966-s8fq2\" (UID: \"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b\") " pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.379922 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.379899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sf8b\" (UniqueName: \"kubernetes.io/projected/6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b-kube-api-access-5sf8b\") pod \"kserve-controller-manager-55b9566966-s8fq2\" (UID: \"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b\") " pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.391582 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.391537    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:17:16.473092 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.473062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsl4g\" (UniqueName: \"kubernetes.io/projected/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-kube-api-access-dsl4g\") pod \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") "
Sep 09 20:17:16.473258 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.473119    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-cert\") pod \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\" (UID: \"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc\") "
Sep 09 20:17:16.475340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.475309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-cert" (OuterVolumeSpecName: "cert") pod "a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" (UID: "a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:16.475340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.475319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-kube-api-access-dsl4g" (OuterVolumeSpecName: "kube-api-access-dsl4g") pod "a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" (UID: "a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc"). InnerVolumeSpecName "kube-api-access-dsl4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:16.508522 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.508479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:16.574499 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.574466    2563 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:17:16.574499 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.574502    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsl4g\" (UniqueName: \"kubernetes.io/projected/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc-kube-api-access-dsl4g\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:17:16.776983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:16.776961    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-s8fq2"]
Sep 09 20:17:16.779428 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:17:16.779400    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc04e79_f455_4c36_9fcf_2db1cc5ebd4b.slice/crio-d484038585f0f8f1c18d79eb04c948689394be13acf5d423fa5ef78faa2ea85f WatchSource:0}: Error finding container d484038585f0f8f1c18d79eb04c948689394be13acf5d423fa5ef78faa2ea85f: Status 404 returned error can't find the container with id d484038585f0f8f1c18d79eb04c948689394be13acf5d423fa5ef78faa2ea85f
Sep 09 20:17:17.089889 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.089857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-s8fq2" event={"ID":"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b","Type":"ContainerStarted","Data":"d484038585f0f8f1c18d79eb04c948689394be13acf5d423fa5ef78faa2ea85f"}
Sep 09 20:17:17.091108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.091086    2563 generic.go:356] "Generic (PLEG): container finished" podID="a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" containerID="5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a" exitCode=0
Sep 09 20:17:17.091198 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.091135    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-5mlvj"
Sep 09 20:17:17.091198 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.091173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-5mlvj" event={"ID":"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc","Type":"ContainerDied","Data":"5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a"}
Sep 09 20:17:17.091268 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.091225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-5mlvj" event={"ID":"a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc","Type":"ContainerDied","Data":"1ff8bd0519fbf9f9c150242bd026447c1d9ab0f9b18f463a21d72183dd04dca5"}
Sep 09 20:17:17.091268 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.091249    2563 scope.go:117] "RemoveContainer" containerID="5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a"
Sep 09 20:17:17.099571 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.099553    2563 scope.go:117] "RemoveContainer" containerID="5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a"
Sep 09 20:17:17.099885 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:17:17.099867    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a\": container with ID starting with 5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a not found: ID does not exist" containerID="5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a"
Sep 09 20:17:17.099951 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.099898    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a"} err="failed to get container status \"5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a\": rpc error: code = NotFound desc = could not find container \"5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a\": container with ID starting with 5a66861b005e028bff5c353360c03887b5fa4c046839f71395906156edc47f1a not found: ID does not exist"
Sep 09 20:17:17.111029 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.111004    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-5mlvj"]
Sep 09 20:17:17.112712 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.112690    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-5mlvj"]
Sep 09 20:17:17.423949 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:17.423877    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" path="/var/lib/kubelet/pods/a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc/volumes"
Sep 09 20:17:18.096821 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:18.096785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-s8fq2" event={"ID":"6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b","Type":"ContainerStarted","Data":"27ea9741f469cce1d4287797b07fdc8dd28e6d16dfbec84e8649385584b09df1"}
Sep 09 20:17:18.097031 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:18.096946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:18.113323 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:18.113278    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-s8fq2" podStartSLOduration=1.713112458 podStartE2EDuration="2.113263797s" totalImagesPullingTime="400.151339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:18.112539731 +0000 UTC m=+365.151709680" watchObservedRunningTime="2026-09-09 20:17:18.113263797 +0000 UTC m=+365.152433786"
Sep 09 20:17:41.641726 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.641699    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8fddf498c-tj64x"]
Sep 09 20:17:41.642188 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.642089    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" containerName="manager"
Sep 09 20:17:41.642188 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.642103    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" containerName="manager"
Sep 09 20:17:41.642188 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.642165    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c840d2-2f12-4a1a-93c2-fc0e94cea4dc" containerName="manager"
Sep 09 20:17:41.644288 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.644268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.657358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.657328    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fddf498c-tj64x"]
Sep 09 20:17:41.789376 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.789339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-service-ca\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.789540 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.789399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-serving-cert\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.789540 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.789449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-oauth-config\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.789540 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.789500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-config\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.789540 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.789526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-trusted-ca-bundle\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.789709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.789586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-oauth-serving-cert\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.789709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.789605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw688\" (UniqueName: \"kubernetes.io/projected/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-kube-api-access-qw688\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.890859 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.890808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-oauth-serving-cert\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.890859 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.890864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw688\" (UniqueName: \"kubernetes.io/projected/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-kube-api-access-qw688\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.890886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-service-ca\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.890928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-serving-cert\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.890959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-oauth-config\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.890987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-config\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.891040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-trusted-ca-bundle\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891638 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.891613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-service-ca\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891784 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.891676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-config\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.891784 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.891691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-oauth-serving-cert\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.892115 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.892094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-trusted-ca-bundle\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.893639 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.893615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-oauth-config\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.893749 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.893683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-console-serving-cert\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.899372 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.899349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw688\" (UniqueName: \"kubernetes.io/projected/80a5acc7-2eaf-4f0d-ba74-a9648d29429d-kube-api-access-qw688\") pod \"console-8fddf498c-tj64x\" (UID: \"80a5acc7-2eaf-4f0d-ba74-a9648d29429d\") " pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:41.953916 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:41.953887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:42.224821 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:42.224782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fddf498c-tj64x"]
Sep 09 20:17:42.227667 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:17:42.227628    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a5acc7_2eaf_4f0d_ba74_a9648d29429d.slice/crio-fae5230ba517a65eeaf9cb91b7a5d92291796797921f3af8669a9a4fd93ca9a5 WatchSource:0}: Error finding container fae5230ba517a65eeaf9cb91b7a5d92291796797921f3af8669a9a4fd93ca9a5: Status 404 returned error can't find the container with id fae5230ba517a65eeaf9cb91b7a5d92291796797921f3af8669a9a4fd93ca9a5
Sep 09 20:17:43.206012 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:43.205977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fddf498c-tj64x" event={"ID":"80a5acc7-2eaf-4f0d-ba74-a9648d29429d","Type":"ContainerStarted","Data":"c055b5c1d10886d210ca91957327d9fc2aee474597190b4994cb07532ebda174"}
Sep 09 20:17:43.206384 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:43.206024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fddf498c-tj64x" event={"ID":"80a5acc7-2eaf-4f0d-ba74-a9648d29429d","Type":"ContainerStarted","Data":"fae5230ba517a65eeaf9cb91b7a5d92291796797921f3af8669a9a4fd93ca9a5"}
Sep 09 20:17:43.224235 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:43.224189    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8fddf498c-tj64x" podStartSLOduration=2.224174837 podStartE2EDuration="2.224174837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:43.222012354 +0000 UTC m=+390.261182305" watchObservedRunningTime="2026-09-09 20:17:43.224174837 +0000 UTC m=+390.263344787"
Sep 09 20:17:49.107076 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:49.107038    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-s8fq2"
Sep 09 20:17:50.388530 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.388501    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-fjssq"]
Sep 09 20:17:50.391247 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.391224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.395541 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.395515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:17:50.395541 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.395530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-flshd\""
Sep 09 20:17:50.395743 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.395593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:17:50.403078 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.403058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-fjssq"]
Sep 09 20:17:50.461020 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.460974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-cert\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.461167 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.461052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqft\" (UniqueName: \"kubernetes.io/projected/d267f173-8274-4c7b-9b83-04c8f25f971a-kube-api-access-jcqft\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.461167 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.461140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-metrics-certs\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.562288 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.562252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-metrics-certs\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.562489 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.562304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-cert\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.562489 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.562340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqft\" (UniqueName: \"kubernetes.io/projected/d267f173-8274-4c7b-9b83-04c8f25f971a-kube-api-access-jcqft\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.562489 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:17:50.562436    2563 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 20:17:50.562705 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:17:50.562515    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-cert podName:d267f173-8274-4c7b-9b83-04c8f25f971a nodeName:}" failed. No retries permitted until 2026-09-09 20:17:51.062493991 +0000 UTC m=+398.101663924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-cert") pod "odh-model-controller-7b966d7984-fjssq" (UID: "d267f173-8274-4c7b-9b83-04c8f25f971a") : secret "odh-model-controller-webhook-cert" not found
Sep 09 20:17:50.564851 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.564831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-metrics-certs\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:50.570932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:50.570906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqft\" (UniqueName: \"kubernetes.io/projected/d267f173-8274-4c7b-9b83-04c8f25f971a-kube-api-access-jcqft\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:51.067606 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:51.067567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-cert\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:51.070125 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:51.070105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d267f173-8274-4c7b-9b83-04c8f25f971a-cert\") pod \"odh-model-controller-7b966d7984-fjssq\" (UID: \"d267f173-8274-4c7b-9b83-04c8f25f971a\") " pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:51.302345 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:51.302322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:51.574356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:51.574312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-fjssq"]
Sep 09 20:17:51.579398 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:17:51.579368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd267f173_8274_4c7b_9b83_04c8f25f971a.slice/crio-2f581d478d4450b8a74614f4ff318ede49185379f1f2745171fd5c5afe8f8728 WatchSource:0}: Error finding container 2f581d478d4450b8a74614f4ff318ede49185379f1f2745171fd5c5afe8f8728: Status 404 returned error can't find the container with id 2f581d478d4450b8a74614f4ff318ede49185379f1f2745171fd5c5afe8f8728
Sep 09 20:17:51.954966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:51.954930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:51.955142 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:51.954996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:51.959560 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:51.959535    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:52.246249 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:52.246152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-fjssq" event={"ID":"d267f173-8274-4c7b-9b83-04c8f25f971a","Type":"ContainerStarted","Data":"2f581d478d4450b8a74614f4ff318ede49185379f1f2745171fd5c5afe8f8728"}
Sep 09 20:17:52.250962 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:52.250931    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8fddf498c-tj64x"
Sep 09 20:17:52.294298 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:52.294266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8874d6dbd-wgtch"]
Sep 09 20:17:55.266558 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:55.266522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-fjssq" event={"ID":"d267f173-8274-4c7b-9b83-04c8f25f971a","Type":"ContainerStarted","Data":"5d53c984ea688ca847f9ec4ab28b63c066c503ab78b282f5e9456e5b3918edf4"}
Sep 09 20:17:55.267026 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:55.266626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:17:55.283099 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:17:55.283053    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-fjssq" podStartSLOduration=2.6480218129999997 podStartE2EDuration="5.283039898s" totalImagesPullingTime="2.635018085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:55.281943324 +0000 UTC m=+402.321113277" watchObservedRunningTime="2026-09-09 20:17:55.283039898 +0000 UTC m=+402.322209858"
Sep 09 20:18:06.274035 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:06.274003    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-fjssq"
Sep 09 20:18:17.324587 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.324521    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8874d6dbd-wgtch" podUID="226e2d19-bbfe-4673-98a5-68ef2d9a7721" containerName="console" containerID="cri-o://76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae" gracePeriod=15
Sep 09 20:18:17.566386 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.566363    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:18:17.700172 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700097    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-serving-cert\") pod \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") "
Sep 09 20:18:17.700172 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z68x6\" (UniqueName: \"kubernetes.io/projected/226e2d19-bbfe-4673-98a5-68ef2d9a7721-kube-api-access-z68x6\") pod \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") "
Sep 09 20:18:17.700172 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-trusted-ca-bundle\") pod \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") "
Sep 09 20:18:17.700442 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700194    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-oauth-serving-cert\") pod \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") "
Sep 09 20:18:17.700442 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-service-ca\") pod \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") "
Sep 09 20:18:17.700442 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-oauth-config\") pod \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") "
Sep 09 20:18:17.700442 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-config\") pod \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\" (UID: \"226e2d19-bbfe-4673-98a5-68ef2d9a7721\") "
Sep 09 20:18:17.700628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700555    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-service-ca" (OuterVolumeSpecName: "service-ca") pod "226e2d19-bbfe-4673-98a5-68ef2d9a7721" (UID: "226e2d19-bbfe-4673-98a5-68ef2d9a7721"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:17.700628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "226e2d19-bbfe-4673-98a5-68ef2d9a7721" (UID: "226e2d19-bbfe-4673-98a5-68ef2d9a7721"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:17.700799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700619    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "226e2d19-bbfe-4673-98a5-68ef2d9a7721" (UID: "226e2d19-bbfe-4673-98a5-68ef2d9a7721"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:17.700799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700741    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:18:17.700799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700758    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-oauth-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:18:17.700799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700773    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:18:17.700986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.700967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-config" (OuterVolumeSpecName: "console-config") pod "226e2d19-bbfe-4673-98a5-68ef2d9a7721" (UID: "226e2d19-bbfe-4673-98a5-68ef2d9a7721"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:17.702453 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.702428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "226e2d19-bbfe-4673-98a5-68ef2d9a7721" (UID: "226e2d19-bbfe-4673-98a5-68ef2d9a7721"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:17.702453 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.702436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226e2d19-bbfe-4673-98a5-68ef2d9a7721-kube-api-access-z68x6" (OuterVolumeSpecName: "kube-api-access-z68x6") pod "226e2d19-bbfe-4673-98a5-68ef2d9a7721" (UID: "226e2d19-bbfe-4673-98a5-68ef2d9a7721"). InnerVolumeSpecName "kube-api-access-z68x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:17.702599 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.702487    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "226e2d19-bbfe-4673-98a5-68ef2d9a7721" (UID: "226e2d19-bbfe-4673-98a5-68ef2d9a7721"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:17.801962 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.801928    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:18:17.801962 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.801960    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:18:17.802161 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.801977    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68x6\" (UniqueName: \"kubernetes.io/projected/226e2d19-bbfe-4673-98a5-68ef2d9a7721-kube-api-access-z68x6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:18:17.802161 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:17.801991    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226e2d19-bbfe-4673-98a5-68ef2d9a7721-console-oauth-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:18:18.371764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.371727    2563 generic.go:356] "Generic (PLEG): container finished" podID="226e2d19-bbfe-4673-98a5-68ef2d9a7721" containerID="76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae" exitCode=0
Sep 09 20:18:18.371764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.371765    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8874d6dbd-wgtch"
Sep 09 20:18:18.372223 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.371818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8874d6dbd-wgtch" event={"ID":"226e2d19-bbfe-4673-98a5-68ef2d9a7721","Type":"ContainerDied","Data":"76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae"}
Sep 09 20:18:18.372223 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.371865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8874d6dbd-wgtch" event={"ID":"226e2d19-bbfe-4673-98a5-68ef2d9a7721","Type":"ContainerDied","Data":"483cb7d32238b89fbb1415e10fe9c370af6b1eefd684619f6125b6a9ccc84da9"}
Sep 09 20:18:18.372223 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.371884    2563 scope.go:117] "RemoveContainer" containerID="76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae"
Sep 09 20:18:18.380615 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.380598    2563 scope.go:117] "RemoveContainer" containerID="76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae"
Sep 09 20:18:18.380904 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:18:18.380884    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae\": container with ID starting with 76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae not found: ID does not exist" containerID="76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae"
Sep 09 20:18:18.380978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.380912    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae"} err="failed to get container status \"76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae\": rpc error: code = NotFound desc = could not find container \"76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae\": container with ID starting with 76f0a9715be85381a397a1a969b22fae04078a02ff1a5ea117195c407dcb2cae not found: ID does not exist"
Sep 09 20:18:18.392596 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.392567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8874d6dbd-wgtch"]
Sep 09 20:18:18.394036 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:18.394019    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8874d6dbd-wgtch"]
Sep 09 20:18:19.428261 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:18:19.428228    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226e2d19-bbfe-4673-98a5-68ef2d9a7721" path="/var/lib/kubelet/pods/226e2d19-bbfe-4673-98a5-68ef2d9a7721/volumes"
Sep 09 20:19:36.761735 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.761702    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-ljngh"]
Sep 09 20:19:36.762200 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.762109    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226e2d19-bbfe-4673-98a5-68ef2d9a7721" containerName="console"
Sep 09 20:19:36.762200 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.762121    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="226e2d19-bbfe-4673-98a5-68ef2d9a7721" containerName="console"
Sep 09 20:19:36.762200 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.762171    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="226e2d19-bbfe-4673-98a5-68ef2d9a7721" containerName="console"
Sep 09 20:19:36.765208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.765191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.768173 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.768150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 09 20:19:36.768277 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.768150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 09 20:19:36.772310 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.772015    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-ljngh"]
Sep 09 20:19:36.848302 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.848268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e234906-6b0e-4421-98bf-efae09647927-data\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.848468 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.848325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/6e234906-6b0e-4421-98bf-efae09647927-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.848468 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.848377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdhz\" (UniqueName: \"kubernetes.io/projected/6e234906-6b0e-4421-98bf-efae09647927-kube-api-access-5bdhz\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.949177 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.949144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/6e234906-6b0e-4421-98bf-efae09647927-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.949304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.949192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdhz\" (UniqueName: \"kubernetes.io/projected/6e234906-6b0e-4421-98bf-efae09647927-kube-api-access-5bdhz\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.949304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.949229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e234906-6b0e-4421-98bf-efae09647927-data\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.949553 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.949535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e234906-6b0e-4421-98bf-efae09647927-data\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.951769 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.951741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/6e234906-6b0e-4421-98bf-efae09647927-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:36.957529 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:36.957499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdhz\" (UniqueName: \"kubernetes.io/projected/6e234906-6b0e-4421-98bf-efae09647927-kube-api-access-5bdhz\") pod \"seaweedfs-tls-custom-8b44d767d-ljngh\" (UID: \"6e234906-6b0e-4421-98bf-efae09647927\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:37.074965 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:37.074938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh"
Sep 09 20:19:37.341256 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:37.341233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-ljngh"]
Sep 09 20:19:37.726195 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:37.726156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh" event={"ID":"6e234906-6b0e-4421-98bf-efae09647927","Type":"ContainerStarted","Data":"2ec605dd1dc2a3cd80e42d03843aef9782a003b5bfec8327119165bdc5b69809"}
Sep 09 20:19:37.726350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:37.726206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh" event={"ID":"6e234906-6b0e-4421-98bf-efae09647927","Type":"ContainerStarted","Data":"d9006773e2ecb4336bc4150480ae54dea45536da55e22ce16711c64b122e12a0"}
Sep 09 20:19:37.742655 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:19:37.742613    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-ljngh" podStartSLOduration=1.480258467 podStartE2EDuration="1.742599321s" totalImagesPullingTime="262.340854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:37.740460774 +0000 UTC m=+504.779630725" watchObservedRunningTime="2026-09-09 20:19:37.742599321 +0000 UTC m=+504.781769273"
Sep 09 20:32:14.350683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.350589    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"]
Sep 09 20:32:14.353987 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.353965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.356339 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.356317    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 09 20:32:14.356561 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.356539    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:32:14.356678 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.356637    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:32:14.356903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.356886    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 09 20:32:14.357715 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.357699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 20:32:14.362429 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.362406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"]
Sep 09 20:32:14.488794 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.488758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b01dc4-9885-41f6-b832-6736315b0980-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.488955 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.488860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b01dc4-9885-41f6-b832-6736315b0980-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.488955 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.488887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cws2p\" (UniqueName: \"kubernetes.io/projected/56b01dc4-9885-41f6-b832-6736315b0980-kube-api-access-cws2p\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.589485 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.589454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b01dc4-9885-41f6-b832-6736315b0980-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.589715 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.589502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws2p\" (UniqueName: \"kubernetes.io/projected/56b01dc4-9885-41f6-b832-6736315b0980-kube-api-access-cws2p\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.589715 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.589564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b01dc4-9885-41f6-b832-6736315b0980-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.590278 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.590259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b01dc4-9885-41f6-b832-6736315b0980-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.592211 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.592188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b01dc4-9885-41f6-b832-6736315b0980-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.597537 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.597513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cws2p\" (UniqueName: \"kubernetes.io/projected/56b01dc4-9885-41f6-b832-6736315b0980-kube-api-access-cws2p\") pod \"message-dumper-predictor-6575bcfdfd-tx84f\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.665177 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.665079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:14.937277 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.937253    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"]
Sep 09 20:32:14.939729 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:32:14.939702    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b01dc4_9885_41f6_b832_6736315b0980.slice/crio-d05bc30be01ce9c5e89a1af3a62f3fd55084309c01edb9783d7980f6c7def1d7 WatchSource:0}: Error finding container d05bc30be01ce9c5e89a1af3a62f3fd55084309c01edb9783d7980f6c7def1d7: Status 404 returned error can't find the container with id d05bc30be01ce9c5e89a1af3a62f3fd55084309c01edb9783d7980f6c7def1d7
Sep 09 20:32:14.941564 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:14.941544    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:32:15.052862 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:15.052820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" event={"ID":"56b01dc4-9885-41f6-b832-6736315b0980","Type":"ContainerStarted","Data":"d05bc30be01ce9c5e89a1af3a62f3fd55084309c01edb9783d7980f6c7def1d7"}
Sep 09 20:32:17.065133 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:17.065097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" event={"ID":"56b01dc4-9885-41f6-b832-6736315b0980","Type":"ContainerStarted","Data":"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"}
Sep 09 20:32:19.077056 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:19.077008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" event={"ID":"56b01dc4-9885-41f6-b832-6736315b0980","Type":"ContainerStarted","Data":"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"}
Sep 09 20:32:19.077487 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:19.077119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:19.077487 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:19.077164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:19.079243 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:19.079218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:32:19.094851 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:19.094788    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" podStartSLOduration=1.477077599 podStartE2EDuration="5.094769939s" totalImagesPullingTime="3.61769234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:19.093075405 +0000 UTC m=+1266.132245382" watchObservedRunningTime="2026-09-09 20:32:19.094769939 +0000 UTC m=+1266.133940268"
Sep 09 20:32:26.090570 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:32:26.090538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:34:29.448237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:29.448208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-tx84f_56b01dc4-9885-41f6-b832-6736315b0980/kserve-container/0.log"
Sep 09 20:34:29.741521 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:29.741435    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"]
Sep 09 20:34:29.741768 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:29.741736    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kserve-container" containerID="cri-o://178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c" gracePeriod=30
Sep 09 20:34:29.741918 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:29.741745    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kube-rbac-proxy" containerID="cri-o://2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb" gracePeriod=30
Sep 09 20:34:29.985866 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:29.985843    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:34:30.029060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.028978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cws2p\" (UniqueName: \"kubernetes.io/projected/56b01dc4-9885-41f6-b832-6736315b0980-kube-api-access-cws2p\") pod \"56b01dc4-9885-41f6-b832-6736315b0980\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") "
Sep 09 20:34:30.029060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.029018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b01dc4-9885-41f6-b832-6736315b0980-kube-rbac-proxy-sar-config\") pod \"56b01dc4-9885-41f6-b832-6736315b0980\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") "
Sep 09 20:34:30.029259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.029088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b01dc4-9885-41f6-b832-6736315b0980-proxy-tls\") pod \"56b01dc4-9885-41f6-b832-6736315b0980\" (UID: \"56b01dc4-9885-41f6-b832-6736315b0980\") "
Sep 09 20:34:30.029399 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.029374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b01dc4-9885-41f6-b832-6736315b0980-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56b01dc4-9885-41f6-b832-6736315b0980" (UID: "56b01dc4-9885-41f6-b832-6736315b0980"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:30.031262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.031239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b01dc4-9885-41f6-b832-6736315b0980-kube-api-access-cws2p" (OuterVolumeSpecName: "kube-api-access-cws2p") pod "56b01dc4-9885-41f6-b832-6736315b0980" (UID: "56b01dc4-9885-41f6-b832-6736315b0980"). InnerVolumeSpecName "kube-api-access-cws2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:30.031380 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.031361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b01dc4-9885-41f6-b832-6736315b0980-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56b01dc4-9885-41f6-b832-6736315b0980" (UID: "56b01dc4-9885-41f6-b832-6736315b0980"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:30.129988 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.129952    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cws2p\" (UniqueName: \"kubernetes.io/projected/56b01dc4-9885-41f6-b832-6736315b0980-kube-api-access-cws2p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:34:30.129988 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.129982    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b01dc4-9885-41f6-b832-6736315b0980-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:34:30.129988 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.129993    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b01dc4-9885-41f6-b832-6736315b0980-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:34:30.667174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.667132    2563 generic.go:356] "Generic (PLEG): container finished" podID="56b01dc4-9885-41f6-b832-6736315b0980" containerID="2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb" exitCode=2
Sep 09 20:34:30.667174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.667157    2563 generic.go:356] "Generic (PLEG): container finished" podID="56b01dc4-9885-41f6-b832-6736315b0980" containerID="178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c" exitCode=2
Sep 09 20:34:30.667579 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.667184    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"
Sep 09 20:34:30.667579 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.667215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" event={"ID":"56b01dc4-9885-41f6-b832-6736315b0980","Type":"ContainerDied","Data":"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"}
Sep 09 20:34:30.667579 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.667281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" event={"ID":"56b01dc4-9885-41f6-b832-6736315b0980","Type":"ContainerDied","Data":"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"}
Sep 09 20:34:30.667579 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.667312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f" event={"ID":"56b01dc4-9885-41f6-b832-6736315b0980","Type":"ContainerDied","Data":"d05bc30be01ce9c5e89a1af3a62f3fd55084309c01edb9783d7980f6c7def1d7"}
Sep 09 20:34:30.667579 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.667351    2563 scope.go:117] "RemoveContainer" containerID="2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"
Sep 09 20:34:30.676072 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.676056    2563 scope.go:117] "RemoveContainer" containerID="178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"
Sep 09 20:34:30.683446 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.683433    2563 scope.go:117] "RemoveContainer" containerID="2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"
Sep 09 20:34:30.683717 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:34:30.683701    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb\": container with ID starting with 2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb not found: ID does not exist" containerID="2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"
Sep 09 20:34:30.683765 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.683724    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"} err="failed to get container status \"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb\": rpc error: code = NotFound desc = could not find container \"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb\": container with ID starting with 2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb not found: ID does not exist"
Sep 09 20:34:30.683765 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.683737    2563 scope.go:117] "RemoveContainer" containerID="178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"
Sep 09 20:34:30.683943 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:34:30.683930    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c\": container with ID starting with 178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c not found: ID does not exist" containerID="178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"
Sep 09 20:34:30.683975 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.683945    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"} err="failed to get container status \"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c\": rpc error: code = NotFound desc = could not find container \"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c\": container with ID starting with 178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c not found: ID does not exist"
Sep 09 20:34:30.683975 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.683957    2563 scope.go:117] "RemoveContainer" containerID="2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"
Sep 09 20:34:30.684161 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.684147    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb"} err="failed to get container status \"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb\": rpc error: code = NotFound desc = could not find container \"2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb\": container with ID starting with 2375e6d85c252cef2c55ae72aaf7d043f40c956cca0561cdd4fcdd124566eafb not found: ID does not exist"
Sep 09 20:34:30.684196 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.684162    2563 scope.go:117] "RemoveContainer" containerID="178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"
Sep 09 20:34:30.684376 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.684361    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c"} err="failed to get container status \"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c\": rpc error: code = NotFound desc = could not find container \"178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c\": container with ID starting with 178383bc0d01b795c8c1157bf75e58dc59314242ad57d5f371d6e1c936ef744c not found: ID does not exist"
Sep 09 20:34:30.688739 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.688718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"]
Sep 09 20:34:30.695113 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:30.695087    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-tx84f"]
Sep 09 20:34:31.424071 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:34:31.424040    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b01dc4-9885-41f6-b832-6736315b0980" path="/var/lib/kubelet/pods/56b01dc4-9885-41f6-b832-6736315b0980/volumes"
Sep 09 20:41:58.172750 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.172712    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"]
Sep 09 20:41:58.173325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.173043    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kserve-container"
Sep 09 20:41:58.173325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.173054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kserve-container"
Sep 09 20:41:58.173325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.173079    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kube-rbac-proxy"
Sep 09 20:41:58.173325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.173085    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kube-rbac-proxy"
Sep 09 20:41:58.173325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.173139    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kserve-container"
Sep 09 20:41:58.173325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.173149    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b01dc4-9885-41f6-b832-6736315b0980" containerName="kube-rbac-proxy"
Sep 09 20:41:58.176366 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.176345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.179282 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.179256    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:41:58.179410 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.179390    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 09 20:41:58.179410 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.179402    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 09 20:41:58.179526 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.179442    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:41:58.180630 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.180613    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:41:58.180735 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.180631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 20:41:58.180735 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.180707    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:41:58.185990 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.185967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"]
Sep 09 20:41:58.198989 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.198567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.199164 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.199036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdf8\" (UniqueName: \"kubernetes.io/projected/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-api-access-4fdf8\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.199164 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.199121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.199277 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.199207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.199328 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.199294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.299727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.299681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.299727 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.299719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdf8\" (UniqueName: \"kubernetes.io/projected/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-api-access-4fdf8\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.299993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.299751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.299993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.299780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.299993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.299957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.300138 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:41:58.300065    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 09 20:41:58.300184 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:41:58.300152    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls podName:fe0d0f3d-5089-40d3-b2ca-3d82574711f5 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:58.800130217 +0000 UTC m=+1845.839300149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-2w8mj" (UID: "fe0d0f3d-5089-40d3-b2ca-3d82574711f5") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 09 20:41:58.300184 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.300150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.300409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.300390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.300528 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.300512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.308670 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.308630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdf8\" (UniqueName: \"kubernetes.io/projected/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-api-access-4fdf8\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.804725 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:58.804684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:58.804904 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:41:58.804804    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 09 20:41:58.804904 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:41:58.804869    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls podName:fe0d0f3d-5089-40d3-b2ca-3d82574711f5 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:59.80485501 +0000 UTC m=+1846.844024938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-2w8mj" (UID: "fe0d0f3d-5089-40d3-b2ca-3d82574711f5") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 09 20:41:59.811601 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:59.811554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:59.814137 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:59.814113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-2w8mj\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:41:59.988139 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:41:59.988093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:42:00.263792 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:00.263760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"]
Sep 09 20:42:00.266339 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:42:00.266305    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0d0f3d_5089_40d3_b2ca_3d82574711f5.slice/crio-078b5d0e124d22c04f59419bf93acf3deeda21eb5931128cd0e6eab89e8e2071 WatchSource:0}: Error finding container 078b5d0e124d22c04f59419bf93acf3deeda21eb5931128cd0e6eab89e8e2071: Status 404 returned error can't find the container with id 078b5d0e124d22c04f59419bf93acf3deeda21eb5931128cd0e6eab89e8e2071
Sep 09 20:42:00.268325 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:00.268312    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:42:00.618592 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:00.618560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerStarted","Data":"078b5d0e124d22c04f59419bf93acf3deeda21eb5931128cd0e6eab89e8e2071"}
Sep 09 20:42:04.640561 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:04.640520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerStarted","Data":"ae519e811ad680962aed638b91172ab2a666561bee17d93570161143f365b079"}
Sep 09 20:42:09.663574 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:09.663541    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerID="ae519e811ad680962aed638b91172ab2a666561bee17d93570161143f365b079" exitCode=0
Sep 09 20:42:09.663973 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:09.663613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerDied","Data":"ae519e811ad680962aed638b91172ab2a666561bee17d93570161143f365b079"}
Sep 09 20:42:14.691820 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:14.691782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerStarted","Data":"9daf118a5256d69e6384c511d61621b2366ff2f6b3a3d721523db12f731424d5"}
Sep 09 20:42:14.692305 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:14.691853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerStarted","Data":"e7bd15d926f43a9f8267557fbdf848e9b823f60515b75bf73bd9f4023ef2f28b"}
Sep 09 20:42:14.692305 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:14.692071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:42:14.712460 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:14.712401    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podStartSLOduration=4.306066118 podStartE2EDuration="16.712381798s" totalImagesPullingTime="7.545234522s" totalInitContainerRuntime="4.861081158s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:14.710451736 +0000 UTC m=+1861.749621687" watchObservedRunningTime="2026-09-09 20:42:14.712381798 +0000 UTC m=+1861.751551748"
Sep 09 20:42:15.702020 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:15.701984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:42:15.703341 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:15.703307    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 20:42:16.706877 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:16.706835    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 20:42:21.711793 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:21.711762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:42:21.712271 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:21.712210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 20:42:31.713606 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:31.713576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:42:49.659539 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:49.659500    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"]
Sep 09 20:42:49.660061 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:49.659830    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kserve-container" containerID="cri-o://e7bd15d926f43a9f8267557fbdf848e9b823f60515b75bf73bd9f4023ef2f28b" gracePeriod=30
Sep 09 20:42:49.660061 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:49.659912    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy" containerID="cri-o://9daf118a5256d69e6384c511d61621b2366ff2f6b3a3d721523db12f731424d5" gracePeriod=30
Sep 09 20:42:49.864041 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:49.864001    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerID="9daf118a5256d69e6384c511d61621b2366ff2f6b3a3d721523db12f731424d5" exitCode=2
Sep 09 20:42:49.864207 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:49.864076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerDied","Data":"9daf118a5256d69e6384c511d61621b2366ff2f6b3a3d721523db12f731424d5"}
Sep 09 20:42:51.707406 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:51.707369    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 09 20:42:54.852275 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.852235    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"]
Sep 09 20:42:54.854783 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.854764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.857523 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.857501    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 09 20:42:54.857856 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.857838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:42:54.864490 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.864465    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"]
Sep 09 20:42:54.875161 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.875130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.875304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.875206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.875304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.875230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/312074d9-a680-465f-ae27-6c3fb555635f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.875387 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.875332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpgp\" (UniqueName: \"kubernetes.io/projected/312074d9-a680-465f-ae27-6c3fb555635f-kube-api-access-fbpgp\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.875387 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.875380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.975861 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.975824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.975861 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.975864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/312074d9-a680-465f-ae27-6c3fb555635f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.976097 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.975917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpgp\" (UniqueName: \"kubernetes.io/projected/312074d9-a680-465f-ae27-6c3fb555635f-kube-api-access-fbpgp\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.976097 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.975938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.976097 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:42:54.975977    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 09 20:42:54.976097 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:42:54.976069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls podName:312074d9-a680-465f-ae27-6c3fb555635f nodeName:}" failed. No retries permitted until 2026-09-09 20:42:55.476047648 +0000 UTC m=+1902.515217598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" (UID: "312074d9-a680-465f-ae27-6c3fb555635f") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 09 20:42:54.976097 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.975985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.976390 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.976361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/312074d9-a680-465f-ae27-6c3fb555635f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.976683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.976639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.976738 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.976721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:54.984845 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:54.984817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpgp\" (UniqueName: \"kubernetes.io/projected/312074d9-a680-465f-ae27-6c3fb555635f-kube-api-access-fbpgp\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:55.478532 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:55.478493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:55.481059 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:55.481034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:55.767595 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:55.767508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:42:56.042104 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:56.042079    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"]
Sep 09 20:42:56.707776 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:56.707730    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 09 20:42:56.900008 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:56.899967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerStarted","Data":"164939366ee46aeb686695c215d1109669d8a1beac3216227b77ba36f1458c3c"}
Sep 09 20:42:56.900188 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:42:56.900025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerStarted","Data":"e3007e7cb1f7976acb07fb4bbb96a6932905fc5aca92fceac62f3536d41f4d9c"}
Sep 09 20:43:00.920098 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:00.920059    2563 generic.go:356] "Generic (PLEG): container finished" podID="312074d9-a680-465f-ae27-6c3fb555635f" containerID="164939366ee46aeb686695c215d1109669d8a1beac3216227b77ba36f1458c3c" exitCode=0
Sep 09 20:43:00.920487 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:00.920113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerDied","Data":"164939366ee46aeb686695c215d1109669d8a1beac3216227b77ba36f1458c3c"}
Sep 09 20:43:01.707844 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.707801    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 09 20:43:01.708029 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.707942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:43:01.927125 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.927087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerStarted","Data":"3a9cf91998bfc90701c50ed419710ddda2d6461ef786bc0a828aca120bf92885"}
Sep 09 20:43:01.927594 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.927152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerStarted","Data":"ea62ed3963fb24539798c305127f3518ea952fab03927d2755666bda2e7484c3"}
Sep 09 20:43:01.927594 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.927501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:43:01.927723 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.927640    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:43:01.928973 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.928948    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 20:43:01.946115 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:01.946055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podStartSLOduration=3.187950705 podStartE2EDuration="7.946037231s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.758086526s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:01.945630531 +0000 UTC m=+1908.984800485" watchObservedRunningTime="2026-09-09 20:43:01.946037231 +0000 UTC m=+1908.985207182"
Sep 09 20:43:02.931696 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:02.931635    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 20:43:03.936248 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:03.936205    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 20:43:06.707628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:06.707578    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 09 20:43:08.940482 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:08.940451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:43:08.941052 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:08.941024    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 20:43:11.707773 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:11.707718    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 09 20:43:16.707360 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:16.707313    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 09 20:43:18.941356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:18.941325    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:43:20.013285 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.013187    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerID="e7bd15d926f43a9f8267557fbdf848e9b823f60515b75bf73bd9f4023ef2f28b" exitCode=137
Sep 09 20:43:20.013672 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.013271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerDied","Data":"e7bd15d926f43a9f8267557fbdf848e9b823f60515b75bf73bd9f4023ef2f28b"}
Sep 09 20:43:20.307665 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.307625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:43:20.383888 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.383857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-cabundle-cert\") pod \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") "
Sep 09 20:43:20.383888 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.383894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls\") pod \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") "
Sep 09 20:43:20.384088 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.383956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kserve-provision-location\") pod \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") "
Sep 09 20:43:20.384088 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.384004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fdf8\" (UniqueName: \"kubernetes.io/projected/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-api-access-4fdf8\") pod \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") "
Sep 09 20:43:20.384088 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.384037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-rbac-proxy-sar-config\") pod \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\" (UID: \"fe0d0f3d-5089-40d3-b2ca-3d82574711f5\") "
Sep 09 20:43:20.384273 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.384245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe0d0f3d-5089-40d3-b2ca-3d82574711f5" (UID: "fe0d0f3d-5089-40d3-b2ca-3d82574711f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:20.384426 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.384404    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe0d0f3d-5089-40d3-b2ca-3d82574711f5" (UID: "fe0d0f3d-5089-40d3-b2ca-3d82574711f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:20.386208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.386176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe0d0f3d-5089-40d3-b2ca-3d82574711f5" (UID: "fe0d0f3d-5089-40d3-b2ca-3d82574711f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:20.386208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.386180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-api-access-4fdf8" (OuterVolumeSpecName: "kube-api-access-4fdf8") pod "fe0d0f3d-5089-40d3-b2ca-3d82574711f5" (UID: "fe0d0f3d-5089-40d3-b2ca-3d82574711f5"). InnerVolumeSpecName "kube-api-access-4fdf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:20.392462 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.392438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe0d0f3d-5089-40d3-b2ca-3d82574711f5" (UID: "fe0d0f3d-5089-40d3-b2ca-3d82574711f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:20.485044 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.485000    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:43:20.485044 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.485041    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:43:20.485044 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.485051    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:43:20.485044 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.485060    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fdf8\" (UniqueName: \"kubernetes.io/projected/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-api-access-4fdf8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:43:20.485281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:20.485068    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe0d0f3d-5089-40d3-b2ca-3d82574711f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:43:21.019601 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.019574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"
Sep 09 20:43:21.019601 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.019589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj" event={"ID":"fe0d0f3d-5089-40d3-b2ca-3d82574711f5","Type":"ContainerDied","Data":"078b5d0e124d22c04f59419bf93acf3deeda21eb5931128cd0e6eab89e8e2071"}
Sep 09 20:43:21.020093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.019678    2563 scope.go:117] "RemoveContainer" containerID="9daf118a5256d69e6384c511d61621b2366ff2f6b3a3d721523db12f731424d5"
Sep 09 20:43:21.028323 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.028306    2563 scope.go:117] "RemoveContainer" containerID="e7bd15d926f43a9f8267557fbdf848e9b823f60515b75bf73bd9f4023ef2f28b"
Sep 09 20:43:21.035964 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.035946    2563 scope.go:117] "RemoveContainer" containerID="ae519e811ad680962aed638b91172ab2a666561bee17d93570161143f365b079"
Sep 09 20:43:21.040780 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.040752    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"]
Sep 09 20:43:21.044853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.044834    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-2w8mj"]
Sep 09 20:43:21.424385 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:21.424353    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" path="/var/lib/kubelet/pods/fe0d0f3d-5089-40d3-b2ca-3d82574711f5/volumes"
Sep 09 20:43:36.174159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:36.174121    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"]
Sep 09 20:43:36.176807 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:36.174367    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container" containerID="cri-o://ea62ed3963fb24539798c305127f3518ea952fab03927d2755666bda2e7484c3" gracePeriod=30
Sep 09 20:43:36.176807 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:36.174413    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy" containerID="cri-o://3a9cf91998bfc90701c50ed419710ddda2d6461ef786bc0a828aca120bf92885" gracePeriod=30
Sep 09 20:43:37.099891 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:37.099856    2563 generic.go:356] "Generic (PLEG): container finished" podID="312074d9-a680-465f-ae27-6c3fb555635f" containerID="3a9cf91998bfc90701c50ed419710ddda2d6461ef786bc0a828aca120bf92885" exitCode=2
Sep 09 20:43:37.099891 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:37.099894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerDied","Data":"3a9cf91998bfc90701c50ed419710ddda2d6461ef786bc0a828aca120bf92885"}
Sep 09 20:43:38.936982 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:38.936937    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 20:43:41.318543 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.318507    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"]
Sep 09 20:43:41.318962 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.318946    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kserve-container"
Sep 09 20:43:41.319012 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.318963    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kserve-container"
Sep 09 20:43:41.319012 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.318984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy"
Sep 09 20:43:41.319012 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.318990    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy"
Sep 09 20:43:41.319012 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.319005    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="storage-initializer"
Sep 09 20:43:41.319012 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.319011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="storage-initializer"
Sep 09 20:43:41.319159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.319065    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kube-rbac-proxy"
Sep 09 20:43:41.319159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.319073    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0d0f3d-5089-40d3-b2ca-3d82574711f5" containerName="kserve-container"
Sep 09 20:43:41.322120 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.322100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.324521 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.324489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 09 20:43:41.324696 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.324548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 09 20:43:41.331218 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.331184    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"]
Sep 09 20:43:41.471585 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.471543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.471585 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.471590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.471822 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.471622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.471822 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.471701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.471822 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.471732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztwtx\" (UniqueName: \"kubernetes.io/projected/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-api-access-ztwtx\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573175 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573175 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztwtx\" (UniqueName: \"kubernetes.io/projected/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-api-access-ztwtx\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573175 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573705 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573896 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.573970 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.573934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.575889 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.575858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.580936 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.580910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztwtx\" (UniqueName: \"kubernetes.io/projected/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-api-access-ztwtx\") pod \"isvc-triton-predictor-5d7db58676-hcg8t\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.633976 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.633944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:43:41.907153 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:41.907115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"]
Sep 09 20:43:41.909447 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:43:41.909417    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a32a81_c03c_4dec_9a54_ffd68197cb5f.slice/crio-a81a73cc95efd4f484c68fc8a697fd30cbce27d1c01f272cc7b640070f804ab3 WatchSource:0}: Error finding container a81a73cc95efd4f484c68fc8a697fd30cbce27d1c01f272cc7b640070f804ab3: Status 404 returned error can't find the container with id a81a73cc95efd4f484c68fc8a697fd30cbce27d1c01f272cc7b640070f804ab3
Sep 09 20:43:42.125156 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:42.125110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerStarted","Data":"96ec8ed77b51589c2d8e2e5a4191c99bd90a2adf63da82602d35f8c35aed1869"}
Sep 09 20:43:42.125342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:42.125188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerStarted","Data":"a81a73cc95efd4f484c68fc8a697fd30cbce27d1c01f272cc7b640070f804ab3"}
Sep 09 20:43:43.937488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:43.937449    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 20:43:46.145598 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:46.145517    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerID="96ec8ed77b51589c2d8e2e5a4191c99bd90a2adf63da82602d35f8c35aed1869" exitCode=0
Sep 09 20:43:46.146009 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:46.145589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerDied","Data":"96ec8ed77b51589c2d8e2e5a4191c99bd90a2adf63da82602d35f8c35aed1869"}
Sep 09 20:43:48.937614 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:48.937135    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 20:43:48.937614 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:48.937294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:43:53.937602 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:53.937560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 20:43:58.936735 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:43:58.936677    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 20:44:03.937220 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:03.937145    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 20:44:06.296872 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:06.296831    2563 generic.go:356] "Generic (PLEG): container finished" podID="312074d9-a680-465f-ae27-6c3fb555635f" containerID="ea62ed3963fb24539798c305127f3518ea952fab03927d2755666bda2e7484c3" exitCode=137
Sep 09 20:44:06.297283 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:06.296885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerDied","Data":"ea62ed3963fb24539798c305127f3518ea952fab03927d2755666bda2e7484c3"}
Sep 09 20:44:06.891668 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:06.891623    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:44:07.031957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.031470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-cabundle-cert\") pod \"312074d9-a680-465f-ae27-6c3fb555635f\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") "
Sep 09 20:44:07.031957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.031553    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/312074d9-a680-465f-ae27-6c3fb555635f-kserve-provision-location\") pod \"312074d9-a680-465f-ae27-6c3fb555635f\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") "
Sep 09 20:44:07.031957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.031599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbpgp\" (UniqueName: \"kubernetes.io/projected/312074d9-a680-465f-ae27-6c3fb555635f-kube-api-access-fbpgp\") pod \"312074d9-a680-465f-ae27-6c3fb555635f\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") "
Sep 09 20:44:07.031957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.031693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls\") pod \"312074d9-a680-465f-ae27-6c3fb555635f\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") "
Sep 09 20:44:07.031957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.031730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-kube-rbac-proxy-sar-config\") pod \"312074d9-a680-465f-ae27-6c3fb555635f\" (UID: \"312074d9-a680-465f-ae27-6c3fb555635f\") "
Sep 09 20:44:07.032356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.031967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "312074d9-a680-465f-ae27-6c3fb555635f" (UID: "312074d9-a680-465f-ae27-6c3fb555635f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:44:07.032356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.032220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "312074d9-a680-465f-ae27-6c3fb555635f" (UID: "312074d9-a680-465f-ae27-6c3fb555635f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:44:07.034770 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.034726    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312074d9-a680-465f-ae27-6c3fb555635f-kube-api-access-fbpgp" (OuterVolumeSpecName: "kube-api-access-fbpgp") pod "312074d9-a680-465f-ae27-6c3fb555635f" (UID: "312074d9-a680-465f-ae27-6c3fb555635f"). InnerVolumeSpecName "kube-api-access-fbpgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:44:07.036058 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.036020    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "312074d9-a680-465f-ae27-6c3fb555635f" (UID: "312074d9-a680-465f-ae27-6c3fb555635f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:44:07.046349 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.046313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312074d9-a680-465f-ae27-6c3fb555635f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "312074d9-a680-465f-ae27-6c3fb555635f" (UID: "312074d9-a680-465f-ae27-6c3fb555635f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:44:07.132787 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.132749    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/312074d9-a680-465f-ae27-6c3fb555635f-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:44:07.132787 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.132783    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:44:07.133045 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.132799    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/312074d9-a680-465f-ae27-6c3fb555635f-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:44:07.133045 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.132813    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/312074d9-a680-465f-ae27-6c3fb555635f-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:44:07.133045 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.132832    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbpgp\" (UniqueName: \"kubernetes.io/projected/312074d9-a680-465f-ae27-6c3fb555635f-kube-api-access-fbpgp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:44:07.307401 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.307361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc" event={"ID":"312074d9-a680-465f-ae27-6c3fb555635f","Type":"ContainerDied","Data":"e3007e7cb1f7976acb07fb4bbb96a6932905fc5aca92fceac62f3536d41f4d9c"}
Sep 09 20:44:07.307935 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.307451    2563 scope.go:117] "RemoveContainer" containerID="3a9cf91998bfc90701c50ed419710ddda2d6461ef786bc0a828aca120bf92885"
Sep 09 20:44:07.307935 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.307379    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"
Sep 09 20:44:07.319511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.319345    2563 scope.go:117] "RemoveContainer" containerID="ea62ed3963fb24539798c305127f3518ea952fab03927d2755666bda2e7484c3"
Sep 09 20:44:07.330963 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.330921    2563 scope.go:117] "RemoveContainer" containerID="164939366ee46aeb686695c215d1109669d8a1beac3216227b77ba36f1458c3c"
Sep 09 20:44:07.337029 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.336972    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"]
Sep 09 20:44:07.340415 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.340367    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qm9cc"]
Sep 09 20:44:07.427354 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:07.427316    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312074d9-a680-465f-ae27-6c3fb555635f" path="/var/lib/kubelet/pods/312074d9-a680-465f-ae27-6c3fb555635f/volumes"
Sep 09 20:44:37.811159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811083    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"]
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="storage-initializer"
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811596    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="storage-initializer"
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811606    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container"
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811612    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container"
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy"
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy"
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811746    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kserve-container"
Sep 09 20:44:37.811760 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.811760    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="312074d9-a680-465f-ae27-6c3fb555635f" containerName="kube-rbac-proxy"
Sep 09 20:44:37.826597 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.826569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"]
Sep 09 20:44:37.826769 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.826734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:37.829311 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.829284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 09 20:44:37.829446 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.829357    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 09 20:44:37.930055 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.930021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:37.930250 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.930098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7125f942-f99a-42bb-ac23-ed4ac135b232-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:37.930250 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.930127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkcg\" (UniqueName: \"kubernetes.io/projected/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-api-access-9dkcg\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:37.930250 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.930150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:37.930250 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:37.930199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7125f942-f99a-42bb-ac23-ed4ac135b232-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.030830 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.030796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7125f942-f99a-42bb-ac23-ed4ac135b232-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.031043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.030910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.031043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.030994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7125f942-f99a-42bb-ac23-ed4ac135b232-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.031043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.031026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkcg\" (UniqueName: \"kubernetes.io/projected/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-api-access-9dkcg\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.031246 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.031050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.031309 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.031237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7125f942-f99a-42bb-ac23-ed4ac135b232-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.031865 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.031841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.031865 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.031856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.034190 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.034172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7125f942-f99a-42bb-ac23-ed4ac135b232-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.040457 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.040435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkcg\" (UniqueName: \"kubernetes.io/projected/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-api-access-9dkcg\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:44:38.139571 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:44:38.139493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:45:01.128424 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:01.128330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"]
Sep 09 20:45:02.043506 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:45:02.043467    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7125f942_f99a_42bb_ac23_ed4ac135b232.slice/crio-6887a495b04fd6fc509a8d952d7ef27d7efe3b7ba7d280f1f2619227cc576099 WatchSource:0}: Error finding container 6887a495b04fd6fc509a8d952d7ef27d7efe3b7ba7d280f1f2619227cc576099: Status 404 returned error can't find the container with id 6887a495b04fd6fc509a8d952d7ef27d7efe3b7ba7d280f1f2619227cc576099
Sep 09 20:45:02.625111 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:02.625063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerStarted","Data":"6887a495b04fd6fc509a8d952d7ef27d7efe3b7ba7d280f1f2619227cc576099"}
Sep 09 20:45:13.684488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:13.684445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerStarted","Data":"51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f"}
Sep 09 20:45:40.825397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:40.825361    2563 generic.go:356] "Generic (PLEG): container finished" podID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerID="51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f" exitCode=0
Sep 09 20:45:40.825816 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:40.825436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerDied","Data":"51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f"}
Sep 09 20:45:41.834592 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:41.834545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerStarted","Data":"1a1d29ce97b6bc81fdc5e64d5bc7712d7ae84824721ae9aa39971c6c9cea2bdc"}
Sep 09 20:45:41.835105 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:41.834628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerStarted","Data":"fae142475592b4f865032bca4ac5d65a2fb2699b89dd368e3ba5f67d6f8dc607"}
Sep 09 20:45:41.835105 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:41.834689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:45:41.853225 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:41.853177    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" podStartSLOduration=1.964159112 podStartE2EDuration="2m0.853154803s" totalImagesPullingTime="1m55.013573413s" totalInitContainerRuntime="3.875422278s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:41.851369797 +0000 UTC m=+2068.890539775" watchObservedRunningTime="2026-09-09 20:45:41.853154803 +0000 UTC m=+2068.892324756"
Sep 09 20:45:42.840612 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:42.840579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:45:42.842164 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:42.842134    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:45:43.846009 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:43.845958    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:45:48.852218 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:48.852187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:45:48.852983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:48.852936    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:45:58.853828 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:45:58.853799    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:46:03.954881 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:03.954839    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"]
Sep 09 20:46:03.955683 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:03.955612    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kserve-container" containerID="cri-o://fae142475592b4f865032bca4ac5d65a2fb2699b89dd368e3ba5f67d6f8dc607" gracePeriod=30
Sep 09 20:46:03.955992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:03.955847    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kube-rbac-proxy" containerID="cri-o://1a1d29ce97b6bc81fdc5e64d5bc7712d7ae84824721ae9aa39971c6c9cea2bdc" gracePeriod=30
Sep 09 20:46:04.959518 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:04.959481    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerID="1a1d29ce97b6bc81fdc5e64d5bc7712d7ae84824721ae9aa39971c6c9cea2bdc" exitCode=2
Sep 09 20:46:04.960123 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:04.959563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerDied","Data":"1a1d29ce97b6bc81fdc5e64d5bc7712d7ae84824721ae9aa39971c6c9cea2bdc"}
Sep 09 20:46:06.972307 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:06.972217    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerID="fae142475592b4f865032bca4ac5d65a2fb2699b89dd368e3ba5f67d6f8dc607" exitCode=0
Sep 09 20:46:06.972307 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:06.972250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerDied","Data":"fae142475592b4f865032bca4ac5d65a2fb2699b89dd368e3ba5f67d6f8dc607"}
Sep 09 20:46:07.358777 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.358748    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:46:07.443293 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.443264    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-rbac-proxy-sar-config\") pod \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") "
Sep 09 20:46:07.443495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.443313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztwtx\" (UniqueName: \"kubernetes.io/projected/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-api-access-ztwtx\") pod \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") "
Sep 09 20:46:07.443495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.443353    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-cabundle-cert\") pod \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") "
Sep 09 20:46:07.443495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.443405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kserve-provision-location\") pod \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") "
Sep 09 20:46:07.443677 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.443497    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-proxy-tls\") pod \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\" (UID: \"b4a32a81-c03c-4dec-9a54-ffd68197cb5f\") "
Sep 09 20:46:07.443794 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.443761    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4a32a81-c03c-4dec-9a54-ffd68197cb5f" (UID: "b4a32a81-c03c-4dec-9a54-ffd68197cb5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:07.443993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.443948    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4a32a81-c03c-4dec-9a54-ffd68197cb5f" (UID: "b4a32a81-c03c-4dec-9a54-ffd68197cb5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:07.444138 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.444106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4a32a81-c03c-4dec-9a54-ffd68197cb5f" (UID: "b4a32a81-c03c-4dec-9a54-ffd68197cb5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:07.446269 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.446238    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4a32a81-c03c-4dec-9a54-ffd68197cb5f" (UID: "b4a32a81-c03c-4dec-9a54-ffd68197cb5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:07.446722 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.446701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-api-access-ztwtx" (OuterVolumeSpecName: "kube-api-access-ztwtx") pod "b4a32a81-c03c-4dec-9a54-ffd68197cb5f" (UID: "b4a32a81-c03c-4dec-9a54-ffd68197cb5f"). InnerVolumeSpecName "kube-api-access-ztwtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:07.544281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.544191    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.544281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.544225    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztwtx\" (UniqueName: \"kubernetes.io/projected/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kube-api-access-ztwtx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.544281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.544234    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.544281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.544243    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.544281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.544252    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a32a81-c03c-4dec-9a54-ffd68197cb5f-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.979501 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.979475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"
Sep 09 20:46:07.979501 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.979483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t" event={"ID":"b4a32a81-c03c-4dec-9a54-ffd68197cb5f","Type":"ContainerDied","Data":"a81a73cc95efd4f484c68fc8a697fd30cbce27d1c01f272cc7b640070f804ab3"}
Sep 09 20:46:07.980055 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.979545    2563 scope.go:117] "RemoveContainer" containerID="1a1d29ce97b6bc81fdc5e64d5bc7712d7ae84824721ae9aa39971c6c9cea2bdc"
Sep 09 20:46:07.988815 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.988796    2563 scope.go:117] "RemoveContainer" containerID="fae142475592b4f865032bca4ac5d65a2fb2699b89dd368e3ba5f67d6f8dc607"
Sep 09 20:46:07.996856 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:07.996838    2563 scope.go:117] "RemoveContainer" containerID="96ec8ed77b51589c2d8e2e5a4191c99bd90a2adf63da82602d35f8c35aed1869"
Sep 09 20:46:08.000910 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:08.000887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"]
Sep 09 20:46:08.004818 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:08.004794    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hcg8t"]
Sep 09 20:46:09.426671 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:46:09.426607    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" path="/var/lib/kubelet/pods/b4a32a81-c03c-4dec-9a54-ffd68197cb5f/volumes"
Sep 09 20:47:45.502889 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:45.502838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerStarted","Data":"9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d"}
Sep 09 20:47:45.503350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:45.503036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:47:45.503350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:45.503077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerStarted","Data":"03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542"}
Sep 09 20:47:45.522224 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:45.522168    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" podStartSLOduration=56.603826216 podStartE2EDuration="3m8.522151327s" totalImagesPullingTime="2m4.023412781s" totalInitContainerRuntime="7.89491233s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:45.520865578 +0000 UTC m=+2192.560035531" watchObservedRunningTime="2026-09-09 20:47:45.522151327 +0000 UTC m=+2192.561321277"
Sep 09 20:47:46.507985 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:46.507941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:47:46.509552 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:46.509513    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:47:47.519418 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:47.519372    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:47:52.524211 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:52.524174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:47:52.524896 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:47:52.524846    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:48:02.526447 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:02.526409    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:48:20.568831 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:20.568798    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"]
Sep 09 20:48:20.569285 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:20.569050    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kserve-container" containerID="cri-o://03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542" gracePeriod=30
Sep 09 20:48:20.569285 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:20.569133    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kube-rbac-proxy" containerID="cri-o://9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d" gracePeriod=30
Sep 09 20:48:21.675337 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:21.675305    2563 generic.go:356] "Generic (PLEG): container finished" podID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerID="9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d" exitCode=2
Sep 09 20:48:21.675337 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:21.675341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerDied","Data":"9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d"}
Sep 09 20:48:22.319376 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.319352    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:48:22.455544 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.455450    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-rbac-proxy-sar-config\") pod \"7125f942-f99a-42bb-ac23-ed4ac135b232\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") "
Sep 09 20:48:22.455544 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.455519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dkcg\" (UniqueName: \"kubernetes.io/projected/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-api-access-9dkcg\") pod \"7125f942-f99a-42bb-ac23-ed4ac135b232\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") "
Sep 09 20:48:22.455786 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.455564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7125f942-f99a-42bb-ac23-ed4ac135b232-proxy-tls\") pod \"7125f942-f99a-42bb-ac23-ed4ac135b232\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") "
Sep 09 20:48:22.455786 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.455584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-cabundle-cert\") pod \"7125f942-f99a-42bb-ac23-ed4ac135b232\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") "
Sep 09 20:48:22.455786 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.455605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7125f942-f99a-42bb-ac23-ed4ac135b232-kserve-provision-location\") pod \"7125f942-f99a-42bb-ac23-ed4ac135b232\" (UID: \"7125f942-f99a-42bb-ac23-ed4ac135b232\") "
Sep 09 20:48:22.456018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.455986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7125f942-f99a-42bb-ac23-ed4ac135b232" (UID: "7125f942-f99a-42bb-ac23-ed4ac135b232"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:22.456018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.455993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7125f942-f99a-42bb-ac23-ed4ac135b232" (UID: "7125f942-f99a-42bb-ac23-ed4ac135b232"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:22.456503 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.456480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7125f942-f99a-42bb-ac23-ed4ac135b232-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7125f942-f99a-42bb-ac23-ed4ac135b232" (UID: "7125f942-f99a-42bb-ac23-ed4ac135b232"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:22.457854 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.457834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-api-access-9dkcg" (OuterVolumeSpecName: "kube-api-access-9dkcg") pod "7125f942-f99a-42bb-ac23-ed4ac135b232" (UID: "7125f942-f99a-42bb-ac23-ed4ac135b232"). InnerVolumeSpecName "kube-api-access-9dkcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:22.457951 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.457855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7125f942-f99a-42bb-ac23-ed4ac135b232-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7125f942-f99a-42bb-ac23-ed4ac135b232" (UID: "7125f942-f99a-42bb-ac23-ed4ac135b232"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:48:22.557356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.557315    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7125f942-f99a-42bb-ac23-ed4ac135b232-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:48:22.557356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.557345    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:48:22.557356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.557355    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7125f942-f99a-42bb-ac23-ed4ac135b232-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:48:22.557356 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.557364    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:48:22.557749 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.557382    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkcg\" (UniqueName: \"kubernetes.io/projected/7125f942-f99a-42bb-ac23-ed4ac135b232-kube-api-access-9dkcg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:48:22.681670 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.681621    2563 generic.go:356] "Generic (PLEG): container finished" podID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerID="03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542" exitCode=0
Sep 09 20:48:22.682145 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.681717    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"
Sep 09 20:48:22.682145 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.681713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerDied","Data":"03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542"}
Sep 09 20:48:22.682145 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.681849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4" event={"ID":"7125f942-f99a-42bb-ac23-ed4ac135b232","Type":"ContainerDied","Data":"6887a495b04fd6fc509a8d952d7ef27d7efe3b7ba7d280f1f2619227cc576099"}
Sep 09 20:48:22.682145 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.681876    2563 scope.go:117] "RemoveContainer" containerID="9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d"
Sep 09 20:48:22.690832 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.690811    2563 scope.go:117] "RemoveContainer" containerID="03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542"
Sep 09 20:48:22.698835 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.698813    2563 scope.go:117] "RemoveContainer" containerID="51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f"
Sep 09 20:48:22.703980 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.703951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"]
Sep 09 20:48:22.707556 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.707535    2563 scope.go:117] "RemoveContainer" containerID="9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d"
Sep 09 20:48:22.707902 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:48:22.707882    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d\": container with ID starting with 9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d not found: ID does not exist" containerID="9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d"
Sep 09 20:48:22.707966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.707912    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d"} err="failed to get container status \"9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d\": rpc error: code = NotFound desc = could not find container \"9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d\": container with ID starting with 9b8ede5f845a32ffeabfbfa5467e9eeb96b7f84d92556a4e363ecad7251d379d not found: ID does not exist"
Sep 09 20:48:22.707966 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.707930    2563 scope.go:117] "RemoveContainer" containerID="03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542"
Sep 09 20:48:22.708201 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:48:22.708183    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542\": container with ID starting with 03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542 not found: ID does not exist" containerID="03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542"
Sep 09 20:48:22.708273 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.708212    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542"} err="failed to get container status \"03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542\": rpc error: code = NotFound desc = could not find container \"03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542\": container with ID starting with 03989b4a10ab18eab409d4550fb9fba5b68799b672f2890ad34c5587bec0a542 not found: ID does not exist"
Sep 09 20:48:22.708273 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.708236    2563 scope.go:117] "RemoveContainer" containerID="51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f"
Sep 09 20:48:22.708482 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:48:22.708467    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f\": container with ID starting with 51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f not found: ID does not exist" containerID="51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f"
Sep 09 20:48:22.708522 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.708487    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f"} err="failed to get container status \"51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f\": rpc error: code = NotFound desc = could not find container \"51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f\": container with ID starting with 51e076acdb44e58fc205bae5ca1abc65213c56ab29ae1e594deaaca77d47c59f not found: ID does not exist"
Sep 09 20:48:22.708603 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:22.708586    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jvkn4"]
Sep 09 20:48:23.424297 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:23.424265    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" path="/var/lib/kubelet/pods/7125f942-f99a-42bb-ac23-ed4ac135b232/volumes"
Sep 09 20:48:25.756725 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.756683    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"]
Sep 09 20:48:25.757215 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757149    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="storage-initializer"
Sep 09 20:48:25.757215 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757165    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="storage-initializer"
Sep 09 20:48:25.757215 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757180    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kserve-container"
Sep 09 20:48:25.757215 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757188    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kserve-container"
Sep 09 20:48:25.757215 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757199    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kube-rbac-proxy"
Sep 09 20:48:25.757215 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757208    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kube-rbac-proxy"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757223    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kserve-container"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757231    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kserve-container"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757261    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="storage-initializer"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757269    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="storage-initializer"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757289    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kube-rbac-proxy"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757296    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kube-rbac-proxy"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757368    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kube-rbac-proxy"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757379    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kserve-container"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757388    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7125f942-f99a-42bb-ac23-ed4ac135b232" containerName="kserve-container"
Sep 09 20:48:25.757402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.757397    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a32a81-c03c-4dec-9a54-ffd68197cb5f" containerName="kube-rbac-proxy"
Sep 09 20:48:25.760619 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.760592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.763440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.763415    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:48:25.763568 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.763488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 20:48:25.763855 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.763840    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:48:25.763922 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.763907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 09 20:48:25.764768 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.764752    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:48:25.764827 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.764764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:48:25.764827 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.764762    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 09 20:48:25.770767 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.770742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"]
Sep 09 20:48:25.782366 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.782342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.782504 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.782382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e548a0e-cd89-47c1-944a-1f6e3765ad02-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.782504 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.782398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9fb\" (UniqueName: \"kubernetes.io/projected/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-api-access-zs9fb\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.782504 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.782485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.782712 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.782536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.883776 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.883737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.883986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.883808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.883986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.883850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.883986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.883897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e548a0e-cd89-47c1-944a-1f6e3765ad02-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.883986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.883920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9fb\" (UniqueName: \"kubernetes.io/projected/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-api-access-zs9fb\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.884286 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.884248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.884532 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.884506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.884639 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.884550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.886409 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.886391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e548a0e-cd89-47c1-944a-1f6e3765ad02-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:25.892867 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:25.892844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9fb\" (UniqueName: \"kubernetes.io/projected/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-api-access-zs9fb\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:26.073141 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:26.073109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:26.345532 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:26.345440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"]
Sep 09 20:48:26.348404 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:48:26.348375    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e548a0e_cd89_47c1_944a_1f6e3765ad02.slice/crio-305be21fbfbb9c99c3c2d10be4514b85a2a0a66b78145024e23798df5e73db29 WatchSource:0}: Error finding container 305be21fbfbb9c99c3c2d10be4514b85a2a0a66b78145024e23798df5e73db29: Status 404 returned error can't find the container with id 305be21fbfbb9c99c3c2d10be4514b85a2a0a66b78145024e23798df5e73db29
Sep 09 20:48:26.702186 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:26.702093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerStarted","Data":"afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c"}
Sep 09 20:48:26.702186 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:26.702155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerStarted","Data":"305be21fbfbb9c99c3c2d10be4514b85a2a0a66b78145024e23798df5e73db29"}
Sep 09 20:48:35.746806 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:35.746716    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerID="afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c" exitCode=0
Sep 09 20:48:35.746806 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:35.746768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerDied","Data":"afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c"}
Sep 09 20:48:36.754512 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:36.754467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerStarted","Data":"5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d"}
Sep 09 20:48:36.755816 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:36.754557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerStarted","Data":"7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68"}
Sep 09 20:48:36.755816 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:36.754817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:36.776633 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:36.776579    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" podStartSLOduration=2.8409700129999997 podStartE2EDuration="11.776563736s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.935593723s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:36.774226784 +0000 UTC m=+2243.813396761" watchObservedRunningTime="2026-09-09 20:48:36.776563736 +0000 UTC m=+2243.815733686"
Sep 09 20:48:37.760999 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:37.760963    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:37.762386 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:37.762360    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 20:48:38.766224 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:38.766171    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 20:48:43.771167 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:43.771140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:48:43.771606 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:43.771546    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 20:48:53.772873 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:48:53.772838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:49:00.970770 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:00.970732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"]
Sep 09 20:49:00.971381 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:00.971027    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kserve-container" containerID="cri-o://7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68" gracePeriod=30
Sep 09 20:49:00.971381 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:00.971183    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kube-rbac-proxy" containerID="cri-o://5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d" gracePeriod=30
Sep 09 20:49:01.882033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:01.881996    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerID="5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d" exitCode=2
Sep 09 20:49:01.882212 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:01.882068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerDied","Data":"5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d"}
Sep 09 20:49:02.511212 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.511183    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:49:02.614159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.614049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-cabundle-cert\") pod \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") "
Sep 09 20:49:02.614159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.614098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kserve-provision-location\") pod \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") "
Sep 09 20:49:02.614159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.614141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-rbac-proxy-sar-config\") pod \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") "
Sep 09 20:49:02.614437 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.614183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e548a0e-cd89-47c1-944a-1f6e3765ad02-proxy-tls\") pod \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") "
Sep 09 20:49:02.614437 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.614231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs9fb\" (UniqueName: \"kubernetes.io/projected/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-api-access-zs9fb\") pod \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\" (UID: \"1e548a0e-cd89-47c1-944a-1f6e3765ad02\") "
Sep 09 20:49:02.614530 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.614508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e548a0e-cd89-47c1-944a-1f6e3765ad02" (UID: "1e548a0e-cd89-47c1-944a-1f6e3765ad02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:02.614575 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.614511    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e548a0e-cd89-47c1-944a-1f6e3765ad02" (UID: "1e548a0e-cd89-47c1-944a-1f6e3765ad02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:02.615123 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.615102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e548a0e-cd89-47c1-944a-1f6e3765ad02" (UID: "1e548a0e-cd89-47c1-944a-1f6e3765ad02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:02.616458 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.616426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e548a0e-cd89-47c1-944a-1f6e3765ad02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e548a0e-cd89-47c1-944a-1f6e3765ad02" (UID: "1e548a0e-cd89-47c1-944a-1f6e3765ad02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:02.616565 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.616479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-api-access-zs9fb" (OuterVolumeSpecName: "kube-api-access-zs9fb") pod "1e548a0e-cd89-47c1-944a-1f6e3765ad02" (UID: "1e548a0e-cd89-47c1-944a-1f6e3765ad02"). InnerVolumeSpecName "kube-api-access-zs9fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:02.715237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.715184    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:49:02.715237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.715229    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e548a0e-cd89-47c1-944a-1f6e3765ad02-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:49:02.715237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.715239    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs9fb\" (UniqueName: \"kubernetes.io/projected/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kube-api-access-zs9fb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:49:02.715237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.715249    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e548a0e-cd89-47c1-944a-1f6e3765ad02-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:49:02.715237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.715258    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e548a0e-cd89-47c1-944a-1f6e3765ad02-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:49:02.888531 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.888445    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerID="7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68" exitCode=0
Sep 09 20:49:02.888531 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.888498    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"
Sep 09 20:49:02.888531 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.888511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerDied","Data":"7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68"}
Sep 09 20:49:02.888781 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.888567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z" event={"ID":"1e548a0e-cd89-47c1-944a-1f6e3765ad02","Type":"ContainerDied","Data":"305be21fbfbb9c99c3c2d10be4514b85a2a0a66b78145024e23798df5e73db29"}
Sep 09 20:49:02.888781 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.888596    2563 scope.go:117] "RemoveContainer" containerID="5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d"
Sep 09 20:49:02.897421 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.897402    2563 scope.go:117] "RemoveContainer" containerID="7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68"
Sep 09 20:49:02.905769 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.905749    2563 scope.go:117] "RemoveContainer" containerID="afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c"
Sep 09 20:49:02.909476 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.909446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"]
Sep 09 20:49:02.914497 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.914473    2563 scope.go:117] "RemoveContainer" containerID="5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d"
Sep 09 20:49:02.914613 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.914595    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s228z"]
Sep 09 20:49:02.914931 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:49:02.914901    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d\": container with ID starting with 5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d not found: ID does not exist" containerID="5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d"
Sep 09 20:49:02.915005 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.914932    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d"} err="failed to get container status \"5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d\": rpc error: code = NotFound desc = could not find container \"5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d\": container with ID starting with 5cb9ca47fcdbb78256f1db4c75a647a0c631aa461c2cc7b7d97bf0ab8a2a931d not found: ID does not exist"
Sep 09 20:49:02.915005 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.914953    2563 scope.go:117] "RemoveContainer" containerID="7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68"
Sep 09 20:49:02.915266 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:49:02.915245    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68\": container with ID starting with 7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68 not found: ID does not exist" containerID="7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68"
Sep 09 20:49:02.915313 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.915277    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68"} err="failed to get container status \"7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68\": rpc error: code = NotFound desc = could not find container \"7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68\": container with ID starting with 7d555c7496d962d306f5e62d39265419709b373aab66e4b02d3050876f695d68 not found: ID does not exist"
Sep 09 20:49:02.915313 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.915294    2563 scope.go:117] "RemoveContainer" containerID="afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c"
Sep 09 20:49:02.915550 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:49:02.915533    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c\": container with ID starting with afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c not found: ID does not exist" containerID="afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c"
Sep 09 20:49:02.915594 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:02.915555    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c"} err="failed to get container status \"afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c\": rpc error: code = NotFound desc = could not find container \"afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c\": container with ID starting with afaecfb90473dc7d9db9e4cf64e68b56a3f4bd90b5cb18fde880d45d1c91a60c not found: ID does not exist"
Sep 09 20:49:03.424812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:03.424780    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" path="/var/lib/kubelet/pods/1e548a0e-cd89-47c1-944a-1f6e3765ad02/volumes"
Sep 09 20:49:06.239554 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.239508    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"]
Sep 09 20:49:06.239959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.239897    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kserve-container"
Sep 09 20:49:06.239959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.239911    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kserve-container"
Sep 09 20:49:06.239959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.239925    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kube-rbac-proxy"
Sep 09 20:49:06.239959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.239931    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kube-rbac-proxy"
Sep 09 20:49:06.239959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.239946    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="storage-initializer"
Sep 09 20:49:06.239959 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.239953    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="storage-initializer"
Sep 09 20:49:06.240151 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.240003    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kserve-container"
Sep 09 20:49:06.240151 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.240011    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e548a0e-cd89-47c1-944a-1f6e3765ad02" containerName="kube-rbac-proxy"
Sep 09 20:49:06.242123 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.242104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.244681 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.244632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:49:06.244882 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.244858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 20:49:06.244980 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.244897    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:49:06.244980 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.244904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:49:06.244980 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.244862    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:49:06.245945 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.245928    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:49:06.246033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.246018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:49:06.252093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.252068    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"]
Sep 09 20:49:06.346003 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.345970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpjm\" (UniqueName: \"kubernetes.io/projected/5824403f-f224-4401-b50d-71f59e489208-kube-api-access-rcpjm\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.346003 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.346013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.346251 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.346056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.346251 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.346149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5824403f-f224-4401-b50d-71f59e489208-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.346251 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.346198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5824403f-f224-4401-b50d-71f59e489208-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.446874 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.446821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.447092 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.446921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.447092 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.446953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5824403f-f224-4401-b50d-71f59e489208-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.447092 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.446987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5824403f-f224-4401-b50d-71f59e489208-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.447092 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.447031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpjm\" (UniqueName: \"kubernetes.io/projected/5824403f-f224-4401-b50d-71f59e489208-kube-api-access-rcpjm\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.447462 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.447426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5824403f-f224-4401-b50d-71f59e489208-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.447699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.447678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.448006 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.447680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.449628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.449608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5824403f-f224-4401-b50d-71f59e489208-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.454894 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.454869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpjm\" (UniqueName: \"kubernetes.io/projected/5824403f-f224-4401-b50d-71f59e489208-kube-api-access-rcpjm\") pod \"isvc-canary-create-predictor-5f7b4999bf-5b98h\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.554060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.554026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:06.826454 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.826426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"]
Sep 09 20:49:06.829011 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:49:06.828984    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5824403f_f224_4401_b50d_71f59e489208.slice/crio-05d321ff309d8c907b27a6dc41c4c8272c2f135dec4e760f9f3a48b7667677a3 WatchSource:0}: Error finding container 05d321ff309d8c907b27a6dc41c4c8272c2f135dec4e760f9f3a48b7667677a3: Status 404 returned error can't find the container with id 05d321ff309d8c907b27a6dc41c4c8272c2f135dec4e760f9f3a48b7667677a3
Sep 09 20:49:06.911903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.911868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerStarted","Data":"3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31"}
Sep 09 20:49:06.912009 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:06.911935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerStarted","Data":"05d321ff309d8c907b27a6dc41c4c8272c2f135dec4e760f9f3a48b7667677a3"}
Sep 09 20:49:11.936955 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:11.936910    2563 generic.go:356] "Generic (PLEG): container finished" podID="5824403f-f224-4401-b50d-71f59e489208" containerID="3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31" exitCode=0
Sep 09 20:49:11.936955 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:11.936954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerDied","Data":"3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31"}
Sep 09 20:49:11.938262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:11.938245    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:49:27.026061 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:27.026026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerStarted","Data":"954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203"}
Sep 09 20:49:27.026469 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:27.026089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerStarted","Data":"1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3"}
Sep 09 20:49:27.026469 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:27.026272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:27.046615 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:27.046567    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podStartSLOduration=2.664062792 podStartE2EDuration="21.046547473s" totalImagesPullingTime="14.349703204s" totalInitContainerRuntime="4.032781477s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:27.044876514 +0000 UTC m=+2294.084046466" watchObservedRunningTime="2026-09-09 20:49:27.046547473 +0000 UTC m=+2294.085717423"
Sep 09 20:49:28.031020 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:28.030985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:28.032527 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:28.032499    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:49:29.035700 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:29.035639    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:49:34.040070 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:34.040039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:49:34.040724 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:34.040694    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:49:44.041357 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:44.041321    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:49:54.041091 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:49:54.041050    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:50:04.041035 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.040987    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:50:04.879687 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.879570    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"]
Sep 09 20:50:04.940792 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.940752    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"]
Sep 09 20:50:04.940952 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.940881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:04.943522 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.943497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 09 20:50:04.943522 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.943501    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:50:04.958133 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.958101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6513a1f2-3055-4a7c-9189-c6b4d2723708-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:04.958254 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.958139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:04.958254 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.958158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:04.958351 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.958269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmkt\" (UniqueName: \"kubernetes.io/projected/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-api-access-lqmkt\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:04.958351 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:04.958303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6513a1f2-3055-4a7c-9189-c6b4d2723708-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.059544 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.059502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6513a1f2-3055-4a7c-9189-c6b4d2723708-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.060033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.059553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.060033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.059575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.060033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.059644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmkt\" (UniqueName: \"kubernetes.io/projected/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-api-access-lqmkt\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.060033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.059700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6513a1f2-3055-4a7c-9189-c6b4d2723708-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.060225 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.060156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6513a1f2-3055-4a7c-9189-c6b4d2723708-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.060431 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.060412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.060635 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.060612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.062535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.062512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6513a1f2-3055-4a7c-9189-c6b4d2723708-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.068276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.068250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmkt\" (UniqueName: \"kubernetes.io/projected/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-api-access-lqmkt\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k9dgj\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.250821 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.250721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:05.523030 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:05.523007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"]
Sep 09 20:50:06.207331 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:06.207293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerStarted","Data":"5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65"}
Sep 09 20:50:06.207830 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:06.207350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerStarted","Data":"b91d71077242e9131b1421115f778773d39cff9ef6499e08f73ee1e3d690e2c3"}
Sep 09 20:50:10.234267 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:10.234231    2563 generic.go:356] "Generic (PLEG): container finished" podID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerID="5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65" exitCode=0
Sep 09 20:50:10.234642 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:10.234279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerDied","Data":"5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65"}
Sep 09 20:50:14.041015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:14.040972    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:50:24.041304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:24.041267    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:50:30.344794 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:30.344753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerStarted","Data":"9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7"}
Sep 09 20:50:30.345188 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:30.344825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerStarted","Data":"cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a"}
Sep 09 20:50:30.345188 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:30.345012    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:30.365765 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:30.365711    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podStartSLOduration=3.058259875 podStartE2EDuration="26.365694201s" totalImagesPullingTime="19.319328187s" totalInitContainerRuntime="3.988106139s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:30.362947609 +0000 UTC m=+2357.402117561" watchObservedRunningTime="2026-09-09 20:50:30.365694201 +0000 UTC m=+2357.404864152"
Sep 09 20:50:31.350776 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:31.350742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:31.352292 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:31.352257    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:50:32.356054 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:32.356011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:50:34.040624 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:34.040585    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:50:37.360457 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:37.360427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:50:37.361033 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:37.361006    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:50:44.041076 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:44.041048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:50:46.242057 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:46.242003    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"]
Sep 09 20:50:46.242520 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:46.242274    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container" containerID="cri-o://1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3" gracePeriod=30
Sep 09 20:50:46.242520 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:46.242305    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kube-rbac-proxy" containerID="cri-o://954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203" gracePeriod=30
Sep 09 20:50:46.422975 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:46.422945    2563 generic.go:356] "Generic (PLEG): container finished" podID="5824403f-f224-4401-b50d-71f59e489208" containerID="954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203" exitCode=2
Sep 09 20:50:46.422975 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:46.422978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerDied","Data":"954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203"}
Sep 09 20:50:47.361103 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:47.361058    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:50:49.036174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:49.036137    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 09 20:50:51.006304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.006276    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:50:51.075702 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.075672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcpjm\" (UniqueName: \"kubernetes.io/projected/5824403f-f224-4401-b50d-71f59e489208-kube-api-access-rcpjm\") pod \"5824403f-f224-4401-b50d-71f59e489208\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") "
Sep 09 20:50:51.075880 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.075734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-kube-rbac-proxy-sar-config\") pod \"5824403f-f224-4401-b50d-71f59e489208\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") "
Sep 09 20:50:51.075880 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.075753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5824403f-f224-4401-b50d-71f59e489208-kserve-provision-location\") pod \"5824403f-f224-4401-b50d-71f59e489208\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") "
Sep 09 20:50:51.075880 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.075797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5824403f-f224-4401-b50d-71f59e489208-proxy-tls\") pod \"5824403f-f224-4401-b50d-71f59e489208\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") "
Sep 09 20:50:51.075880 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.075816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-cabundle-cert\") pod \"5824403f-f224-4401-b50d-71f59e489208\" (UID: \"5824403f-f224-4401-b50d-71f59e489208\") "
Sep 09 20:50:51.076135 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.076106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5824403f-f224-4401-b50d-71f59e489208" (UID: "5824403f-f224-4401-b50d-71f59e489208"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:51.076135 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.076119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5824403f-f224-4401-b50d-71f59e489208-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5824403f-f224-4401-b50d-71f59e489208" (UID: "5824403f-f224-4401-b50d-71f59e489208"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:51.076222 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.076204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5824403f-f224-4401-b50d-71f59e489208" (UID: "5824403f-f224-4401-b50d-71f59e489208"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:51.078102 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.078080    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5824403f-f224-4401-b50d-71f59e489208-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5824403f-f224-4401-b50d-71f59e489208" (UID: "5824403f-f224-4401-b50d-71f59e489208"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:51.078102 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.078086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5824403f-f224-4401-b50d-71f59e489208-kube-api-access-rcpjm" (OuterVolumeSpecName: "kube-api-access-rcpjm") pod "5824403f-f224-4401-b50d-71f59e489208" (UID: "5824403f-f224-4401-b50d-71f59e489208"). InnerVolumeSpecName "kube-api-access-rcpjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:51.177318 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.177284    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5824403f-f224-4401-b50d-71f59e489208-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.177318 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.177311    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.177318 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.177322    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcpjm\" (UniqueName: \"kubernetes.io/projected/5824403f-f224-4401-b50d-71f59e489208-kube-api-access-rcpjm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.177517 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.177330    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5824403f-f224-4401-b50d-71f59e489208-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.177517 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.177340    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5824403f-f224-4401-b50d-71f59e489208-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.446734 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.446701    2563 generic.go:356] "Generic (PLEG): container finished" podID="5824403f-f224-4401-b50d-71f59e489208" containerID="1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3" exitCode=0
Sep 09 20:50:51.446872 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.446762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerDied","Data":"1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3"}
Sep 09 20:50:51.446872 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.446770    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"
Sep 09 20:50:51.446872 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.446825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h" event={"ID":"5824403f-f224-4401-b50d-71f59e489208","Type":"ContainerDied","Data":"05d321ff309d8c907b27a6dc41c4c8272c2f135dec4e760f9f3a48b7667677a3"}
Sep 09 20:50:51.446872 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.446862    2563 scope.go:117] "RemoveContainer" containerID="954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203"
Sep 09 20:50:51.456178 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.456157    2563 scope.go:117] "RemoveContainer" containerID="1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3"
Sep 09 20:50:51.464422 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.464402    2563 scope.go:117] "RemoveContainer" containerID="3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31"
Sep 09 20:50:51.472175 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.472156    2563 scope.go:117] "RemoveContainer" containerID="954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203"
Sep 09 20:50:51.472422 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:50:51.472399    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203\": container with ID starting with 954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203 not found: ID does not exist" containerID="954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203"
Sep 09 20:50:51.472528 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.472430    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203"} err="failed to get container status \"954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203\": rpc error: code = NotFound desc = could not find container \"954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203\": container with ID starting with 954fb941d737e87dc05dfca1022fe0e07333ddf9da78206638a16fc445495203 not found: ID does not exist"
Sep 09 20:50:51.472528 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.472447    2563 scope.go:117] "RemoveContainer" containerID="1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3"
Sep 09 20:50:51.472716 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:50:51.472699    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3\": container with ID starting with 1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3 not found: ID does not exist" containerID="1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3"
Sep 09 20:50:51.472770 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.472721    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3"} err="failed to get container status \"1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3\": rpc error: code = NotFound desc = could not find container \"1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3\": container with ID starting with 1d639b2048eb30f89d85afe7456ec4d8f66cb5c7997ed2d7f04c396b04e17de3 not found: ID does not exist"
Sep 09 20:50:51.472770 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.472740    2563 scope.go:117] "RemoveContainer" containerID="3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31"
Sep 09 20:50:51.472992 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:50:51.472973    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31\": container with ID starting with 3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31 not found: ID does not exist" containerID="3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31"
Sep 09 20:50:51.473041 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.472997    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31"} err="failed to get container status \"3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31\": rpc error: code = NotFound desc = could not find container \"3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31\": container with ID starting with 3b6377434c546324f2814b3ac8c3db7b6abf89353afd16e0eaf157f20fce8a31 not found: ID does not exist"
Sep 09 20:50:51.480061 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.480039    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"]
Sep 09 20:50:51.485483 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.485463    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-5b98h"]
Sep 09 20:50:51.492988 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.492971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"]
Sep 09 20:50:51.493330 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493317    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="storage-initializer"
Sep 09 20:50:51.493378 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493331    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="storage-initializer"
Sep 09 20:50:51.493378 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493356    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container"
Sep 09 20:50:51.493378 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493365    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container"
Sep 09 20:50:51.493378 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493377    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kube-rbac-proxy"
Sep 09 20:50:51.493535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493385    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kube-rbac-proxy"
Sep 09 20:50:51.493535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493457    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kserve-container"
Sep 09 20:50:51.493535 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.493468    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5824403f-f224-4401-b50d-71f59e489208" containerName="kube-rbac-proxy"
Sep 09 20:50:51.498257 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.498242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.501171 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.501155    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:50:51.501262 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.501205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:50:51.506448 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.506427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"]
Sep 09 20:50:51.574166 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.574136    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"]
Sep 09 20:50:51.577738 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.577716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.580500 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.580480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:50:51.580762 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.580741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.580853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.580769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21ebb998-97fd-44d5-bb1b-83162665b98c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.580853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.580795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.580853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.580849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21ebb998-97fd-44d5-bb1b-83162665b98c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.581000 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.580887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmbp\" (UniqueName: \"kubernetes.io/projected/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-api-access-xqmbp\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.588365 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.588337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"]
Sep 09 20:50:51.681708 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.681708 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21ebb998-97fd-44d5-bb1b-83162665b98c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.681708 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56xr\" (UniqueName: \"kubernetes.io/projected/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-api-access-t56xr\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.682018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.682018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21ebb998-97fd-44d5-bb1b-83162665b98c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.682018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7271526f-16f7-4515-a2e4-2ef96bc66c91-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.682018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmbp\" (UniqueName: \"kubernetes.io/projected/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-api-access-xqmbp\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.682018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.682018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.682018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.681940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.682340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.682199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21ebb998-97fd-44d5-bb1b-83162665b98c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.682340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.682257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.682407 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.682344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.684322 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.684303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21ebb998-97fd-44d5-bb1b-83162665b98c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.689936 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.689914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmbp\" (UniqueName: \"kubernetes.io/projected/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-api-access-xqmbp\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:51.783253 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.783212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56xr\" (UniqueName: \"kubernetes.io/projected/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-api-access-t56xr\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.783426 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.783270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7271526f-16f7-4515-a2e4-2ef96bc66c91-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.783426 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.783320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.783426 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.783345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.783426 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.783379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.783598 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:50:51.783469    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:50:51.783598 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:50:51.783557    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls podName:7271526f-16f7-4515-a2e4-2ef96bc66c91 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:52.283534477 +0000 UTC m=+2379.322704419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls") pod "isvc-canary-promote-predictor-648d7cfc8f-z6t5b" (UID: "7271526f-16f7-4515-a2e4-2ef96bc66c91") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:50:51.783825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.783803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7271526f-16f7-4515-a2e4-2ef96bc66c91-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.784070 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.784054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.784107 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.784072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.792359 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.792339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56xr\" (UniqueName: \"kubernetes.io/projected/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-api-access-t56xr\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:51.809376 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:51.809346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:52.088110 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:52.088084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"]
Sep 09 20:50:52.090355 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:50:52.090320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ebb998_97fd_44d5_bb1b_83162665b98c.slice/crio-620079d98d9a6a2e51c15e74241f986e5427718147c883b22de6ad496a994142 WatchSource:0}: Error finding container 620079d98d9a6a2e51c15e74241f986e5427718147c883b22de6ad496a994142: Status 404 returned error can't find the container with id 620079d98d9a6a2e51c15e74241f986e5427718147c883b22de6ad496a994142
Sep 09 20:50:52.287531 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:52.287489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:52.290210 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:52.290178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-z6t5b\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:52.454570 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:52.454477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerStarted","Data":"543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4"}
Sep 09 20:50:52.454570 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:52.454544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerStarted","Data":"620079d98d9a6a2e51c15e74241f986e5427718147c883b22de6ad496a994142"}
Sep 09 20:50:52.491616 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:52.491586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:52.779166 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:52.779138    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"]
Sep 09 20:50:52.781098 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:50:52.781067    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7271526f_16f7_4515_a2e4_2ef96bc66c91.slice/crio-79a6d9820459bcbbd99c6f14f613f49749ae173fbc25f187ff3d75680744ad6a WatchSource:0}: Error finding container 79a6d9820459bcbbd99c6f14f613f49749ae173fbc25f187ff3d75680744ad6a: Status 404 returned error can't find the container with id 79a6d9820459bcbbd99c6f14f613f49749ae173fbc25f187ff3d75680744ad6a
Sep 09 20:50:53.426015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:53.425982    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5824403f-f224-4401-b50d-71f59e489208" path="/var/lib/kubelet/pods/5824403f-f224-4401-b50d-71f59e489208/volumes"
Sep 09 20:50:53.461684 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:53.461625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerStarted","Data":"a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676"}
Sep 09 20:50:53.461837 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:53.461723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerStarted","Data":"79a6d9820459bcbbd99c6f14f613f49749ae173fbc25f187ff3d75680744ad6a"}
Sep 09 20:50:56.483100 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:56.483017    2563 generic.go:356] "Generic (PLEG): container finished" podID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerID="543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4" exitCode=0
Sep 09 20:50:56.483495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:56.483097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerDied","Data":"543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4"}
Sep 09 20:50:57.361062 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:57.361023    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:50:57.491933 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:57.491894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerStarted","Data":"da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0"}
Sep 09 20:50:57.492357 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:57.491967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerStarted","Data":"d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c"}
Sep 09 20:50:57.492357 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:57.492161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:57.493665 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:57.493624    2563 generic.go:356] "Generic (PLEG): container finished" podID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerID="a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676" exitCode=0
Sep 09 20:50:57.493747 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:57.493700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerDied","Data":"a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676"}
Sep 09 20:50:57.510995 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:57.510947    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podStartSLOduration=2.547144077 podStartE2EDuration="6.51093054s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.963786463s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:57.508849599 +0000 UTC m=+2384.548019562" watchObservedRunningTime="2026-09-09 20:50:57.51093054 +0000 UTC m=+2384.550100588"
Sep 09 20:50:58.501380 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.501346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerStarted","Data":"91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473"}
Sep 09 20:50:58.501874 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.501407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerStarted","Data":"93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19"}
Sep 09 20:50:58.501874 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.501556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:50:58.501874 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.501850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:58.502037 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.501993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:50:58.503146 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.503121    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:50:58.503227 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.503125    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:50:58.520061 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:58.520009    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podStartSLOduration=3.763164652 podStartE2EDuration="7.519991183s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756826531s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:58.519536106 +0000 UTC m=+2385.558706058" watchObservedRunningTime="2026-09-09 20:50:58.519991183 +0000 UTC m=+2385.559161135"
Sep 09 20:50:59.512216 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:59.512172    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:50:59.512662 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:50:59.512349    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:51:04.517499 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:04.517411    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:51:04.518040 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:04.518011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:51:04.518281 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:04.518261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:51:04.518698 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:04.518679    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:51:07.361082 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:07.361037    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:51:14.518739 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:14.518701    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:51:14.519171 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:14.518710    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:51:17.361024 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:17.360981    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:51:24.518493 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:24.518453    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:51:24.518889 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:24.518590    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:51:27.360910 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:27.360871    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:51:34.518340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:34.518251    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:51:34.518767 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:34.518698    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:51:37.362307 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:37.362271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:51:44.518059 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:44.518020    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:51:44.519276 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:44.519247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:51:44.957684 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:44.957622    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"]
Sep 09 20:51:44.957969 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:44.957925    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" containerID="cri-o://cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a" gracePeriod=30
Sep 09 20:51:44.958046 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:44.957960    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kube-rbac-proxy" containerID="cri-o://9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7" gracePeriod=30
Sep 09 20:51:45.734616 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:45.734583    2563 generic.go:356] "Generic (PLEG): container finished" podID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerID="9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7" exitCode=2
Sep 09 20:51:45.735007 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:45.734667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerDied","Data":"9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7"}
Sep 09 20:51:47.357043 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:47.357004    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 09 20:51:47.361396 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:47.361364    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:51:48.712757 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.712731    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:51:48.757618 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.757586    2563 generic.go:356] "Generic (PLEG): container finished" podID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerID="cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a" exitCode=0
Sep 09 20:51:48.757825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.757637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerDied","Data":"cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a"}
Sep 09 20:51:48.757825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.757690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"
Sep 09 20:51:48.757825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.757728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj" event={"ID":"6513a1f2-3055-4a7c-9189-c6b4d2723708","Type":"ContainerDied","Data":"b91d71077242e9131b1421115f778773d39cff9ef6499e08f73ee1e3d690e2c3"}
Sep 09 20:51:48.757825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.757756    2563 scope.go:117] "RemoveContainer" containerID="9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7"
Sep 09 20:51:48.766214 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.766196    2563 scope.go:117] "RemoveContainer" containerID="cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a"
Sep 09 20:51:48.774056 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.774039    2563 scope.go:117] "RemoveContainer" containerID="5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65"
Sep 09 20:51:48.781716 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.781696    2563 scope.go:117] "RemoveContainer" containerID="9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7"
Sep 09 20:51:48.781992 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:51:48.781974    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7\": container with ID starting with 9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7 not found: ID does not exist" containerID="9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7"
Sep 09 20:51:48.782040 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.782002    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7"} err="failed to get container status \"9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7\": rpc error: code = NotFound desc = could not find container \"9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7\": container with ID starting with 9f1843f28e4adc6e8c1e388fc374b120ea3d4e2a3f55419fcd23f067a52e66a7 not found: ID does not exist"
Sep 09 20:51:48.782040 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.782025    2563 scope.go:117] "RemoveContainer" containerID="cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a"
Sep 09 20:51:48.782295 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:51:48.782274    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a\": container with ID starting with cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a not found: ID does not exist" containerID="cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a"
Sep 09 20:51:48.782347 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.782302    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a"} err="failed to get container status \"cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a\": rpc error: code = NotFound desc = could not find container \"cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a\": container with ID starting with cca96ca72abce6c9115a85861036c411beb4e1fb9aec1b654cc5eec7262c362a not found: ID does not exist"
Sep 09 20:51:48.782347 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.782318    2563 scope.go:117] "RemoveContainer" containerID="5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65"
Sep 09 20:51:48.782543 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:51:48.782529    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65\": container with ID starting with 5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65 not found: ID does not exist" containerID="5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65"
Sep 09 20:51:48.782584 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.782546    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65"} err="failed to get container status \"5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65\": rpc error: code = NotFound desc = could not find container \"5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65\": container with ID starting with 5f2cef8131cf0f577f57d20a0bcdf86ab852bb5b3180395c6c1643eb47a62b65 not found: ID does not exist"
Sep 09 20:51:48.792872 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.792845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6513a1f2-3055-4a7c-9189-c6b4d2723708-kserve-provision-location\") pod \"6513a1f2-3055-4a7c-9189-c6b4d2723708\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") "
Sep 09 20:51:48.792973 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.792885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqmkt\" (UniqueName: \"kubernetes.io/projected/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-api-access-lqmkt\") pod \"6513a1f2-3055-4a7c-9189-c6b4d2723708\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") "
Sep 09 20:51:48.792973 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.792921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-rbac-proxy-sar-config\") pod \"6513a1f2-3055-4a7c-9189-c6b4d2723708\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") "
Sep 09 20:51:48.792973 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.792967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6513a1f2-3055-4a7c-9189-c6b4d2723708-proxy-tls\") pod \"6513a1f2-3055-4a7c-9189-c6b4d2723708\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") "
Sep 09 20:51:48.793135 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.793005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-cabundle-cert\") pod \"6513a1f2-3055-4a7c-9189-c6b4d2723708\" (UID: \"6513a1f2-3055-4a7c-9189-c6b4d2723708\") "
Sep 09 20:51:48.793269 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.793243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6513a1f2-3055-4a7c-9189-c6b4d2723708-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6513a1f2-3055-4a7c-9189-c6b4d2723708" (UID: "6513a1f2-3055-4a7c-9189-c6b4d2723708"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:48.793332 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.793306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6513a1f2-3055-4a7c-9189-c6b4d2723708" (UID: "6513a1f2-3055-4a7c-9189-c6b4d2723708"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:48.793377 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.793336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6513a1f2-3055-4a7c-9189-c6b4d2723708" (UID: "6513a1f2-3055-4a7c-9189-c6b4d2723708"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:48.795023 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.795004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-api-access-lqmkt" (OuterVolumeSpecName: "kube-api-access-lqmkt") pod "6513a1f2-3055-4a7c-9189-c6b4d2723708" (UID: "6513a1f2-3055-4a7c-9189-c6b4d2723708"). InnerVolumeSpecName "kube-api-access-lqmkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:48.795120 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.795072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6513a1f2-3055-4a7c-9189-c6b4d2723708-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6513a1f2-3055-4a7c-9189-c6b4d2723708" (UID: "6513a1f2-3055-4a7c-9189-c6b4d2723708"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:51:48.894236 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.894195    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:51:48.894236 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.894226    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6513a1f2-3055-4a7c-9189-c6b4d2723708-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:51:48.894236 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.894236    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmkt\" (UniqueName: \"kubernetes.io/projected/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-api-access-lqmkt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:51:48.894236 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.894245    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6513a1f2-3055-4a7c-9189-c6b4d2723708-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:51:48.894517 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:48.894254    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6513a1f2-3055-4a7c-9189-c6b4d2723708-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:51:49.081210 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:49.081172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"]
Sep 09 20:51:49.084909 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:49.084878    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k9dgj"]
Sep 09 20:51:49.425473 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:49.425439    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" path="/var/lib/kubelet/pods/6513a1f2-3055-4a7c-9189-c6b4d2723708/volumes"
Sep 09 20:51:54.518606 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:54.518561    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:51:54.519017 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:51:54.518743    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:52:04.518581 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:04.518535    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:52:04.518993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:04.518711    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:52:14.519441 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:14.519405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:52:14.519964 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:14.519947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:52:21.494743 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:21.494709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"]
Sep 09 20:52:21.495309 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:21.495031    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" containerID="cri-o://93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19" gracePeriod=30
Sep 09 20:52:21.495309 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:21.495090    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kube-rbac-proxy" containerID="cri-o://91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473" gracePeriod=30
Sep 09 20:52:21.912626 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:21.912592    2563 generic.go:356] "Generic (PLEG): container finished" podID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerID="91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473" exitCode=2
Sep 09 20:52:21.912806 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:21.912677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerDied","Data":"91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473"}
Sep 09 20:52:24.512885 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:24.512844    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 09 20:52:24.518690 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:24.518642    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:52:26.341279 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.341254    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:52:26.419248 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419150    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls\") pod \"7271526f-16f7-4515-a2e4-2ef96bc66c91\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") "
Sep 09 20:52:26.419248 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-rbac-proxy-sar-config\") pod \"7271526f-16f7-4515-a2e4-2ef96bc66c91\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") "
Sep 09 20:52:26.419487 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7271526f-16f7-4515-a2e4-2ef96bc66c91-kserve-provision-location\") pod \"7271526f-16f7-4515-a2e4-2ef96bc66c91\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") "
Sep 09 20:52:26.419487 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t56xr\" (UniqueName: \"kubernetes.io/projected/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-api-access-t56xr\") pod \"7271526f-16f7-4515-a2e4-2ef96bc66c91\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") "
Sep 09 20:52:26.419487 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-cabundle-cert\") pod \"7271526f-16f7-4515-a2e4-2ef96bc66c91\" (UID: \"7271526f-16f7-4515-a2e4-2ef96bc66c91\") "
Sep 09 20:52:26.419687 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419562    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7271526f-16f7-4515-a2e4-2ef96bc66c91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7271526f-16f7-4515-a2e4-2ef96bc66c91" (UID: "7271526f-16f7-4515-a2e4-2ef96bc66c91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:26.419687 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7271526f-16f7-4515-a2e4-2ef96bc66c91" (UID: "7271526f-16f7-4515-a2e4-2ef96bc66c91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:52:26.419820 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419798    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7271526f-16f7-4515-a2e4-2ef96bc66c91" (UID: "7271526f-16f7-4515-a2e4-2ef96bc66c91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:52:26.419820 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419813    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:52:26.420071 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.419828    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7271526f-16f7-4515-a2e4-2ef96bc66c91-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:52:26.421502 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.421481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7271526f-16f7-4515-a2e4-2ef96bc66c91" (UID: "7271526f-16f7-4515-a2e4-2ef96bc66c91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:52:26.421607 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.421538    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-api-access-t56xr" (OuterVolumeSpecName: "kube-api-access-t56xr") pod "7271526f-16f7-4515-a2e4-2ef96bc66c91" (UID: "7271526f-16f7-4515-a2e4-2ef96bc66c91"). InnerVolumeSpecName "kube-api-access-t56xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:26.520986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.520949    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56xr\" (UniqueName: \"kubernetes.io/projected/7271526f-16f7-4515-a2e4-2ef96bc66c91-kube-api-access-t56xr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:52:26.520986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.520979    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7271526f-16f7-4515-a2e4-2ef96bc66c91-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:52:26.520986 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.520988    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7271526f-16f7-4515-a2e4-2ef96bc66c91-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:52:26.937574 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.937538    2563 generic.go:356] "Generic (PLEG): container finished" podID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerID="93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19" exitCode=0
Sep 09 20:52:26.937764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.937595    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"
Sep 09 20:52:26.937764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.937604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerDied","Data":"93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19"}
Sep 09 20:52:26.937764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.937681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b" event={"ID":"7271526f-16f7-4515-a2e4-2ef96bc66c91","Type":"ContainerDied","Data":"79a6d9820459bcbbd99c6f14f613f49749ae173fbc25f187ff3d75680744ad6a"}
Sep 09 20:52:26.937764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.937715    2563 scope.go:117] "RemoveContainer" containerID="91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473"
Sep 09 20:52:26.946771 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.946750    2563 scope.go:117] "RemoveContainer" containerID="93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19"
Sep 09 20:52:26.954640 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.954620    2563 scope.go:117] "RemoveContainer" containerID="a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676"
Sep 09 20:52:26.959724 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.959701    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"]
Sep 09 20:52:26.962839 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.962813    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-z6t5b"]
Sep 09 20:52:26.963342 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.963328    2563 scope.go:117] "RemoveContainer" containerID="91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473"
Sep 09 20:52:26.963678 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:52:26.963632    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473\": container with ID starting with 91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473 not found: ID does not exist" containerID="91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473"
Sep 09 20:52:26.963738 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.963689    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473"} err="failed to get container status \"91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473\": rpc error: code = NotFound desc = could not find container \"91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473\": container with ID starting with 91ec5bbb3de3d1b478c56c3088219db73b610c009b23938e0b0293a595016473 not found: ID does not exist"
Sep 09 20:52:26.963738 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.963708    2563 scope.go:117] "RemoveContainer" containerID="93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19"
Sep 09 20:52:26.963996 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:52:26.963972    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19\": container with ID starting with 93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19 not found: ID does not exist" containerID="93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19"
Sep 09 20:52:26.964070 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.964005    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19"} err="failed to get container status \"93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19\": rpc error: code = NotFound desc = could not find container \"93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19\": container with ID starting with 93a7423fa10353186f590954b6e9405a44fcb074bcc9cbb9fea69dfea320aa19 not found: ID does not exist"
Sep 09 20:52:26.964070 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.964029    2563 scope.go:117] "RemoveContainer" containerID="a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676"
Sep 09 20:52:26.964279 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:52:26.964261    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676\": container with ID starting with a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676 not found: ID does not exist" containerID="a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676"
Sep 09 20:52:26.964328 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:26.964285    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676"} err="failed to get container status \"a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676\": rpc error: code = NotFound desc = could not find container \"a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676\": container with ID starting with a7aa6df010cd86693d337f58a2cc87e99694d018c8dc7c8c444725a4ea864676 not found: ID does not exist"
Sep 09 20:52:27.425559 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:27.425517    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" path="/var/lib/kubelet/pods/7271526f-16f7-4515-a2e4-2ef96bc66c91/volumes"
Sep 09 20:52:55.385640 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.385603    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"]
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386136    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="storage-initializer"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386158    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="storage-initializer"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386180    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386189    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386203    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kube-rbac-proxy"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386213    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kube-rbac-proxy"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386230    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="storage-initializer"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386239    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="storage-initializer"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386251    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kube-rbac-proxy"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386258    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kube-rbac-proxy"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386285    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container"
Sep 09 20:52:55.386321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386293    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container"
Sep 09 20:52:55.387015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386378    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kube-rbac-proxy"
Sep 09 20:52:55.387015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386392    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6513a1f2-3055-4a7c-9189-c6b4d2723708" containerName="kserve-container"
Sep 09 20:52:55.387015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386403    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kserve-container"
Sep 09 20:52:55.387015 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.386412    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7271526f-16f7-4515-a2e4-2ef96bc66c91" containerName="kube-rbac-proxy"
Sep 09 20:52:55.389860 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.389838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.392141 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.392119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 09 20:52:55.392259 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.392187    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:52:55.399132 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.399111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"]
Sep 09 20:52:55.571898 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.571851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5l8f\" (UniqueName: \"kubernetes.io/projected/d3660280-718c-4a95-bce8-3e8397496975-kube-api-access-j5l8f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.571898 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.571902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3660280-718c-4a95-bce8-3e8397496975-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.572132 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.571988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.572132 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.572018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3660280-718c-4a95-bce8-3e8397496975-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.572132 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.572066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.672764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.672675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5l8f\" (UniqueName: \"kubernetes.io/projected/d3660280-718c-4a95-bce8-3e8397496975-kube-api-access-j5l8f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.672764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.672715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3660280-718c-4a95-bce8-3e8397496975-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.672764 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.672752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.673075 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.672777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3660280-718c-4a95-bce8-3e8397496975-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.673075 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.672794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.673269 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.673244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3660280-718c-4a95-bce8-3e8397496975-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.673505 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.673484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.673574 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.673507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.675414 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.675397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3660280-718c-4a95-bce8-3e8397496975-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.680497 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.680476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5l8f\" (UniqueName: \"kubernetes.io/projected/d3660280-718c-4a95-bce8-3e8397496975-kube-api-access-j5l8f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5t788\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.700540 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.700519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:52:55.973973 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:55.973897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"]
Sep 09 20:52:55.977243 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:52:55.977217    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3660280_718c_4a95_bce8_3e8397496975.slice/crio-355c7f92d2c7a12d9479652952329dc771ecf285af3ce122a5fefbe53115bb65 WatchSource:0}: Error finding container 355c7f92d2c7a12d9479652952329dc771ecf285af3ce122a5fefbe53115bb65: Status 404 returned error can't find the container with id 355c7f92d2c7a12d9479652952329dc771ecf285af3ce122a5fefbe53115bb65
Sep 09 20:52:56.069268 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:56.069235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerStarted","Data":"ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4"}
Sep 09 20:52:56.069411 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:52:56.069295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerStarted","Data":"355c7f92d2c7a12d9479652952329dc771ecf285af3ce122a5fefbe53115bb65"}
Sep 09 20:53:00.088958 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:00.088924    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3660280-718c-4a95-bce8-3e8397496975" containerID="ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4" exitCode=0
Sep 09 20:53:00.089385 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:00.088985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerDied","Data":"ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4"}
Sep 09 20:53:01.101863 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:01.101823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerStarted","Data":"3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a"}
Sep 09 20:53:01.102291 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:01.101909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerStarted","Data":"87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1"}
Sep 09 20:53:01.102291 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:01.102130    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:53:01.102291 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:01.102177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:53:01.103672 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:01.103635    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:01.122440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:01.122394    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podStartSLOduration=2.209669543 podStartE2EDuration="6.122380268s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.912710725s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:01.119833138 +0000 UTC m=+2508.159003090" watchObservedRunningTime="2026-09-09 20:53:01.122380268 +0000 UTC m=+2508.161550218"
Sep 09 20:53:02.106957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:02.106912    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:07.112173 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:07.112095    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:53:07.112604 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:07.112577    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:17.113510 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:17.113470    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:27.113319 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:27.113281    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:37.112536 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:37.112496    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:47.113337 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:47.113294    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:51.651914 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:51.651873    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"]
Sep 09 20:53:51.652377 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:51.652326    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" containerID="cri-o://d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c" gracePeriod=30
Sep 09 20:53:51.652377 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:51.652337    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kube-rbac-proxy" containerID="cri-o://da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0" gracePeriod=30
Sep 09 20:53:52.337164 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:52.337132    2563 generic.go:356] "Generic (PLEG): container finished" podID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerID="da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0" exitCode=2
Sep 09 20:53:52.337337 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:52.337179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerDied","Data":"da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0"}
Sep 09 20:53:54.512376 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:54.512325    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 20:53:54.518708 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:54.518682    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:53:56.511217 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.511190    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:53:56.693221 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-rbac-proxy-sar-config\") pod \"21ebb998-97fd-44d5-bb1b-83162665b98c\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") "
Sep 09 20:53:56.693221 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21ebb998-97fd-44d5-bb1b-83162665b98c-kserve-provision-location\") pod \"21ebb998-97fd-44d5-bb1b-83162665b98c\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") "
Sep 09 20:53:56.693503 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-cabundle-cert\") pod \"21ebb998-97fd-44d5-bb1b-83162665b98c\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") "
Sep 09 20:53:56.693503 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21ebb998-97fd-44d5-bb1b-83162665b98c-proxy-tls\") pod \"21ebb998-97fd-44d5-bb1b-83162665b98c\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") "
Sep 09 20:53:56.693503 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqmbp\" (UniqueName: \"kubernetes.io/projected/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-api-access-xqmbp\") pod \"21ebb998-97fd-44d5-bb1b-83162665b98c\" (UID: \"21ebb998-97fd-44d5-bb1b-83162665b98c\") "
Sep 09 20:53:56.693717 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693499    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ebb998-97fd-44d5-bb1b-83162665b98c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21ebb998-97fd-44d5-bb1b-83162665b98c" (UID: "21ebb998-97fd-44d5-bb1b-83162665b98c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:56.693717 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21ebb998-97fd-44d5-bb1b-83162665b98c" (UID: "21ebb998-97fd-44d5-bb1b-83162665b98c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:56.693717 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693607    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21ebb998-97fd-44d5-bb1b-83162665b98c-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.693717 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.693635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21ebb998-97fd-44d5-bb1b-83162665b98c" (UID: "21ebb998-97fd-44d5-bb1b-83162665b98c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:56.695604 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.695575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ebb998-97fd-44d5-bb1b-83162665b98c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21ebb998-97fd-44d5-bb1b-83162665b98c" (UID: "21ebb998-97fd-44d5-bb1b-83162665b98c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:56.695930 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.695894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-api-access-xqmbp" (OuterVolumeSpecName: "kube-api-access-xqmbp") pod "21ebb998-97fd-44d5-bb1b-83162665b98c" (UID: "21ebb998-97fd-44d5-bb1b-83162665b98c"). InnerVolumeSpecName "kube-api-access-xqmbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:56.794572 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.794538    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.794572 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.794571    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21ebb998-97fd-44d5-bb1b-83162665b98c-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.794838 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.794583    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmbp\" (UniqueName: \"kubernetes.io/projected/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-api-access-xqmbp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.794838 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.794598    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21ebb998-97fd-44d5-bb1b-83162665b98c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.837838 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.837801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"]
Sep 09 20:53:56.838352 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838336    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="storage-initializer"
Sep 09 20:53:56.838398 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838357    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="storage-initializer"
Sep 09 20:53:56.838398 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838382    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kube-rbac-proxy"
Sep 09 20:53:56.838398 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838391    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kube-rbac-proxy"
Sep 09 20:53:56.838489 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838406    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container"
Sep 09 20:53:56.838489 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838415    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container"
Sep 09 20:53:56.838550 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838500    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kserve-container"
Sep 09 20:53:56.838550 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.838513    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerName="kube-rbac-proxy"
Sep 09 20:53:56.842049 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.842022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:56.844367 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.844345    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 09 20:53:56.844545 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.844525    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:53:56.850228 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.850178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"]
Sep 09 20:53:56.912512 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.912470    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"]
Sep 09 20:53:56.916037 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.916017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:56.918689 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.918642    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:53:56.926087 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.926059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"]
Sep 09 20:53:56.996513 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.996424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b03402cb-18a3-463f-a6c6-45b5cda892d5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:56.996513 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.996478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:56.996513 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.996504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:56.996789 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.996551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfsx\" (UniqueName: \"kubernetes.io/projected/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-api-access-hcfsx\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:56.996789 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:56.996633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03402cb-18a3-463f-a6c6-45b5cda892d5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.097495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.097495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhfh\" (UniqueName: \"kubernetes.io/projected/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-api-access-zmhfh\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.097495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.097788 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03402cb-18a3-463f-a6c6-45b5cda892d5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.097788 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.097788 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b03402cb-18a3-463f-a6c6-45b5cda892d5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.097788 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.097992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.097992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.097992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.097973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfsx\" (UniqueName: \"kubernetes.io/projected/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-api-access-hcfsx\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.098196 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.098176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b03402cb-18a3-463f-a6c6-45b5cda892d5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.098504 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.098481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.098551 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.098512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.100201 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.100178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03402cb-18a3-463f-a6c6-45b5cda892d5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.105739 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.105714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfsx\" (UniqueName: \"kubernetes.io/projected/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-api-access-hcfsx\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.114042 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.114002    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:53:57.155143 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.155104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:53:57.198453 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.198411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.198453 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.198455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhfh\" (UniqueName: \"kubernetes.io/projected/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-api-access-zmhfh\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.198720 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.198474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.198720 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.198549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.198720 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.198601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.199093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.199068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.199311 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.199292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.199374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.199350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.200983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.200966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.208304 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.208275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhfh\" (UniqueName: \"kubernetes.io/projected/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-api-access-zmhfh\") pod \"isvc-canary-rollback-predictor-6477dbf789-bht9z\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.227023 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.226982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:53:57.364081 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.364048    2563 generic.go:356] "Generic (PLEG): container finished" podID="21ebb998-97fd-44d5-bb1b-83162665b98c" containerID="d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c" exitCode=0
Sep 09 20:53:57.364236 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.364119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"
Sep 09 20:53:57.364236 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.364139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerDied","Data":"d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c"}
Sep 09 20:53:57.364308 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.364233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj" event={"ID":"21ebb998-97fd-44d5-bb1b-83162665b98c","Type":"ContainerDied","Data":"620079d98d9a6a2e51c15e74241f986e5427718147c883b22de6ad496a994142"}
Sep 09 20:53:57.364308 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.364264    2563 scope.go:117] "RemoveContainer" containerID="da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0"
Sep 09 20:53:57.373885 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.373783    2563 scope.go:117] "RemoveContainer" containerID="d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c"
Sep 09 20:53:57.381800 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.381781    2563 scope.go:117] "RemoveContainer" containerID="543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4"
Sep 09 20:53:57.386534 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.386505    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"]
Sep 09 20:53:57.390248 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.390230    2563 scope.go:117] "RemoveContainer" containerID="da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0"
Sep 09 20:53:57.390560 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:53:57.390536    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0\": container with ID starting with da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0 not found: ID does not exist" containerID="da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0"
Sep 09 20:53:57.390662 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.390566    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0"} err="failed to get container status \"da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0\": rpc error: code = NotFound desc = could not find container \"da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0\": container with ID starting with da0d5ac127ec5867461e1f1f0882726e7167d4cdec479e05fb920a73a70d71e0 not found: ID does not exist"
Sep 09 20:53:57.390662 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.390584    2563 scope.go:117] "RemoveContainer" containerID="d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c"
Sep 09 20:53:57.390854 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:53:57.390836    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c\": container with ID starting with d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c not found: ID does not exist" containerID="d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c"
Sep 09 20:53:57.390896 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.390862    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c"} err="failed to get container status \"d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c\": rpc error: code = NotFound desc = could not find container \"d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c\": container with ID starting with d791559eed6526f5d123dc60759fef3ff4c28f0bf907b0950ffeedfcd3d4206c not found: ID does not exist"
Sep 09 20:53:57.390896 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.390880    2563 scope.go:117] "RemoveContainer" containerID="543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4"
Sep 09 20:53:57.391171 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:53:57.391146    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4\": container with ID starting with 543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4 not found: ID does not exist" containerID="543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4"
Sep 09 20:53:57.391261 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.391181    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4"} err="failed to get container status \"543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4\": rpc error: code = NotFound desc = could not find container \"543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4\": container with ID starting with 543bf6df835258a7d9f1345a559918d8d6298280f490a2b227f2631f00dca3f4 not found: ID does not exist"
Sep 09 20:53:57.394200 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.394177    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-5jvvj"]
Sep 09 20:53:57.425159 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.425128    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ebb998-97fd-44d5-bb1b-83162665b98c" path="/var/lib/kubelet/pods/21ebb998-97fd-44d5-bb1b-83162665b98c/volumes"
Sep 09 20:53:57.435570 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.435528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"]
Sep 09 20:53:57.439513 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:53:57.439484    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03402cb_18a3_463f_a6c6_45b5cda892d5.slice/crio-0048aa9ebaa0a6d039e916cb48077d6b1067d7d51cbb770371af259e2f86cb3f WatchSource:0}: Error finding container 0048aa9ebaa0a6d039e916cb48077d6b1067d7d51cbb770371af259e2f86cb3f: Status 404 returned error can't find the container with id 0048aa9ebaa0a6d039e916cb48077d6b1067d7d51cbb770371af259e2f86cb3f
Sep 09 20:53:57.506162 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:57.506137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"]
Sep 09 20:53:57.508772 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:53:57.508742    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc9de2f_0502_4f34_bb13_d2d6f16ed3f8.slice/crio-60774ee3200887302f61f14265a2863ca29aa6d44752cbb7d0f7c111bac4a565 WatchSource:0}: Error finding container 60774ee3200887302f61f14265a2863ca29aa6d44752cbb7d0f7c111bac4a565: Status 404 returned error can't find the container with id 60774ee3200887302f61f14265a2863ca29aa6d44752cbb7d0f7c111bac4a565
Sep 09 20:53:58.370385 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:58.370343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerStarted","Data":"79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908"}
Sep 09 20:53:58.370873 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:58.370402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerStarted","Data":"60774ee3200887302f61f14265a2863ca29aa6d44752cbb7d0f7c111bac4a565"}
Sep 09 20:53:58.371903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:58.371881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerStarted","Data":"0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53"}
Sep 09 20:53:58.371965 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:53:58.371936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerStarted","Data":"0048aa9ebaa0a6d039e916cb48077d6b1067d7d51cbb770371af259e2f86cb3f"}
Sep 09 20:54:01.388660 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:01.388623    2563 generic.go:356] "Generic (PLEG): container finished" podID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerID="0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53" exitCode=0
Sep 09 20:54:01.389059 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:01.388666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerDied","Data":"0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53"}
Sep 09 20:54:02.394515 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.394479    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerID="79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908" exitCode=0
Sep 09 20:54:02.395047 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.394564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerDied","Data":"79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908"}
Sep 09 20:54:02.396833 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.396813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerStarted","Data":"a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184"}
Sep 09 20:54:02.396929 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.396860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerStarted","Data":"80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554"}
Sep 09 20:54:02.397048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.397029    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:54:02.397104 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.397073    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:54:02.398511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.398487    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:02.432541 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:02.432453    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podStartSLOduration=2.761700051 podStartE2EDuration="6.432430879s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670730828s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:02.431611155 +0000 UTC m=+2569.470781106" watchObservedRunningTime="2026-09-09 20:54:02.432430879 +0000 UTC m=+2569.471600833"
Sep 09 20:54:03.403605 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:03.403566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerStarted","Data":"7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086"}
Sep 09 20:54:03.404226 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:03.403640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerStarted","Data":"ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182"}
Sep 09 20:54:03.404226 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:03.403890    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:03.404226 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:03.404109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:54:03.404426 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:03.404229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:54:03.405511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:03.405490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:54:03.421831 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:03.421781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podStartSLOduration=3.591258014 podStartE2EDuration="7.421766126s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.830508112s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:03.42073249 +0000 UTC m=+2570.459902437" watchObservedRunningTime="2026-09-09 20:54:03.421766126 +0000 UTC m=+2570.460936129"
Sep 09 20:54:04.408030 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:04.407986    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:54:07.113854 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:07.113816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:54:08.408328 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:08.408297    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:54:08.408908 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:08.408880    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:09.412463 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:09.412379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:54:09.412950 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:09.412925    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:54:15.503957 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:15.503924    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"]
Sep 09 20:54:15.504366 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:15.504171    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" containerID="cri-o://87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1" gracePeriod=30
Sep 09 20:54:15.504366 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:15.504207    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kube-rbac-proxy" containerID="cri-o://3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a" gracePeriod=30
Sep 09 20:54:16.464680 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:16.464630    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3660280-718c-4a95-bce8-3e8397496975" containerID="3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a" exitCode=2
Sep 09 20:54:16.464868 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:16.464706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerDied","Data":"3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a"}
Sep 09 20:54:17.107532 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:17.107489    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 09 20:54:17.112853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:17.112830    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 20:54:18.408757 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:18.408721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:19.369715 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.369690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:54:19.413784 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.413693    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:54:19.480449 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.480410    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3660280-718c-4a95-bce8-3e8397496975" containerID="87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1" exitCode=0
Sep 09 20:54:19.480664 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.480468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerDied","Data":"87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1"}
Sep 09 20:54:19.480664 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.480471    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"
Sep 09 20:54:19.480664 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.480519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788" event={"ID":"d3660280-718c-4a95-bce8-3e8397496975","Type":"ContainerDied","Data":"355c7f92d2c7a12d9479652952329dc771ecf285af3ce122a5fefbe53115bb65"}
Sep 09 20:54:19.480664 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.480544    2563 scope.go:117] "RemoveContainer" containerID="3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a"
Sep 09 20:54:19.487039 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3660280-718c-4a95-bce8-3e8397496975-proxy-tls\") pod \"d3660280-718c-4a95-bce8-3e8397496975\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") "
Sep 09 20:54:19.487172 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5l8f\" (UniqueName: \"kubernetes.io/projected/d3660280-718c-4a95-bce8-3e8397496975-kube-api-access-j5l8f\") pod \"d3660280-718c-4a95-bce8-3e8397496975\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") "
Sep 09 20:54:19.487231 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3660280-718c-4a95-bce8-3e8397496975-kserve-provision-location\") pod \"d3660280-718c-4a95-bce8-3e8397496975\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") "
Sep 09 20:54:19.487231 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487199    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-kube-rbac-proxy-sar-config\") pod \"d3660280-718c-4a95-bce8-3e8397496975\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") "
Sep 09 20:54:19.487321 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-cabundle-cert\") pod \"d3660280-718c-4a95-bce8-3e8397496975\" (UID: \"d3660280-718c-4a95-bce8-3e8397496975\") "
Sep 09 20:54:19.487638 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3660280-718c-4a95-bce8-3e8397496975-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3660280-718c-4a95-bce8-3e8397496975" (UID: "d3660280-718c-4a95-bce8-3e8397496975"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:19.487916 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487893    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3660280-718c-4a95-bce8-3e8397496975" (UID: "d3660280-718c-4a95-bce8-3e8397496975"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:19.487992 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.487970    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3660280-718c-4a95-bce8-3e8397496975" (UID: "d3660280-718c-4a95-bce8-3e8397496975"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:19.489374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.489351    2563 scope.go:117] "RemoveContainer" containerID="87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1"
Sep 09 20:54:19.489478 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.489433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3660280-718c-4a95-bce8-3e8397496975-kube-api-access-j5l8f" (OuterVolumeSpecName: "kube-api-access-j5l8f") pod "d3660280-718c-4a95-bce8-3e8397496975" (UID: "d3660280-718c-4a95-bce8-3e8397496975"). InnerVolumeSpecName "kube-api-access-j5l8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:19.489793 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.489761    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3660280-718c-4a95-bce8-3e8397496975-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3660280-718c-4a95-bce8-3e8397496975" (UID: "d3660280-718c-4a95-bce8-3e8397496975"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:19.505401 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.505380    2563 scope.go:117] "RemoveContainer" containerID="ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4"
Sep 09 20:54:19.513631 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.513607    2563 scope.go:117] "RemoveContainer" containerID="3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a"
Sep 09 20:54:19.513949 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:54:19.513927    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a\": container with ID starting with 3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a not found: ID does not exist" containerID="3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a"
Sep 09 20:54:19.514000 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.513960    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a"} err="failed to get container status \"3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a\": rpc error: code = NotFound desc = could not find container \"3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a\": container with ID starting with 3dca786327318311f554b5c895ab094a168ac4816d6bf4084a3ce01f71caeb3a not found: ID does not exist"
Sep 09 20:54:19.514000 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.513979    2563 scope.go:117] "RemoveContainer" containerID="87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1"
Sep 09 20:54:19.514257 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:54:19.514237    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1\": container with ID starting with 87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1 not found: ID does not exist" containerID="87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1"
Sep 09 20:54:19.514307 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.514263    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1"} err="failed to get container status \"87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1\": rpc error: code = NotFound desc = could not find container \"87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1\": container with ID starting with 87bf0387de5d6125b64fcdb5a413ba9d1f3d856435bae861565a290ae7d066e1 not found: ID does not exist"
Sep 09 20:54:19.514307 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.514285    2563 scope.go:117] "RemoveContainer" containerID="ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4"
Sep 09 20:54:19.514516 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:54:19.514500    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4\": container with ID starting with ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4 not found: ID does not exist" containerID="ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4"
Sep 09 20:54:19.514557 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.514521    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4"} err="failed to get container status \"ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4\": rpc error: code = NotFound desc = could not find container \"ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4\": container with ID starting with ddd67785873a4ca507e80f3576a0fe79e9727a559b689e752c2669974aa677c4 not found: ID does not exist"
Sep 09 20:54:19.588420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.588380    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:54:19.588420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.588411    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3660280-718c-4a95-bce8-3e8397496975-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:54:19.588420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.588421    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3660280-718c-4a95-bce8-3e8397496975-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:54:19.588420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.588430    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5l8f\" (UniqueName: \"kubernetes.io/projected/d3660280-718c-4a95-bce8-3e8397496975-kube-api-access-j5l8f\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:54:19.588732 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.588438    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3660280-718c-4a95-bce8-3e8397496975-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:54:19.803539 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.803505    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"]
Sep 09 20:54:19.807415 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:19.807390    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5t788"]
Sep 09 20:54:21.424592 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:21.424560    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3660280-718c-4a95-bce8-3e8397496975" path="/var/lib/kubelet/pods/d3660280-718c-4a95-bce8-3e8397496975/volumes"
Sep 09 20:54:28.409073 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:28.409029    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:29.412900 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:29.412856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:54:38.409093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:38.409050    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:39.413686 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:39.413623    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:54:48.409263 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:48.409223    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:49.412863 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:49.412826    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:54:58.408790 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:58.408746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:54:59.413286 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:54:59.413251    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:55:08.409499 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:08.409460    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:55:09.413574 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:09.413485    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:55:18.409866 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:18.409829    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:55:19.413799 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:19.413769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:55:26.850674 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:26.850616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"]
Sep 09 20:55:26.851128 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:26.850903    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" containerID="cri-o://80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554" gracePeriod=30
Sep 09 20:55:26.851128 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:26.851002    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kube-rbac-proxy" containerID="cri-o://a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184" gracePeriod=30
Sep 09 20:55:27.795590 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:27.795556    2563 generic.go:356] "Generic (PLEG): container finished" podID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerID="a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184" exitCode=2
Sep 09 20:55:27.795789 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:27.795629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerDied","Data":"a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184"}
Sep 09 20:55:28.405157 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:28.405105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 09 20:55:28.409706 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:28.409676    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:55:31.706528 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.706503    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:55:31.809398 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-rbac-proxy-sar-config\") pod \"b03402cb-18a3-463f-a6c6-45b5cda892d5\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") "
Sep 09 20:55:31.809552 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcfsx\" (UniqueName: \"kubernetes.io/projected/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-api-access-hcfsx\") pod \"b03402cb-18a3-463f-a6c6-45b5cda892d5\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") "
Sep 09 20:55:31.809552 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809448    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03402cb-18a3-463f-a6c6-45b5cda892d5-proxy-tls\") pod \"b03402cb-18a3-463f-a6c6-45b5cda892d5\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") "
Sep 09 20:55:31.809552 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b03402cb-18a3-463f-a6c6-45b5cda892d5-kserve-provision-location\") pod \"b03402cb-18a3-463f-a6c6-45b5cda892d5\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") "
Sep 09 20:55:31.809750 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-cabundle-cert\") pod \"b03402cb-18a3-463f-a6c6-45b5cda892d5\" (UID: \"b03402cb-18a3-463f-a6c6-45b5cda892d5\") "
Sep 09 20:55:31.809811 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809752    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b03402cb-18a3-463f-a6c6-45b5cda892d5" (UID: "b03402cb-18a3-463f-a6c6-45b5cda892d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:31.809952 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03402cb-18a3-463f-a6c6-45b5cda892d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b03402cb-18a3-463f-a6c6-45b5cda892d5" (UID: "b03402cb-18a3-463f-a6c6-45b5cda892d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:55:31.809952 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.809941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b03402cb-18a3-463f-a6c6-45b5cda892d5" (UID: "b03402cb-18a3-463f-a6c6-45b5cda892d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:31.811794 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.811777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03402cb-18a3-463f-a6c6-45b5cda892d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b03402cb-18a3-463f-a6c6-45b5cda892d5" (UID: "b03402cb-18a3-463f-a6c6-45b5cda892d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:31.811882 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.811865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-api-access-hcfsx" (OuterVolumeSpecName: "kube-api-access-hcfsx") pod "b03402cb-18a3-463f-a6c6-45b5cda892d5" (UID: "b03402cb-18a3-463f-a6c6-45b5cda892d5"). InnerVolumeSpecName "kube-api-access-hcfsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:31.820784 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.820753    2563 generic.go:356] "Generic (PLEG): container finished" podID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerID="80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554" exitCode=0
Sep 09 20:55:31.820886 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.820808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"
Sep 09 20:55:31.820886 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.820818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerDied","Data":"80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554"}
Sep 09 20:55:31.820886 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.820869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7" event={"ID":"b03402cb-18a3-463f-a6c6-45b5cda892d5","Type":"ContainerDied","Data":"0048aa9ebaa0a6d039e916cb48077d6b1067d7d51cbb770371af259e2f86cb3f"}
Sep 09 20:55:31.820993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.820909    2563 scope.go:117] "RemoveContainer" containerID="a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184"
Sep 09 20:55:31.829948 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.829930    2563 scope.go:117] "RemoveContainer" containerID="80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554"
Sep 09 20:55:31.837812 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.837793    2563 scope.go:117] "RemoveContainer" containerID="0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53"
Sep 09 20:55:31.844009 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.843987    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"]
Sep 09 20:55:31.846436 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.846419    2563 scope.go:117] "RemoveContainer" containerID="a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184"
Sep 09 20:55:31.846733 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:55:31.846714    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184\": container with ID starting with a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184 not found: ID does not exist" containerID="a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184"
Sep 09 20:55:31.846814 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.846741    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184"} err="failed to get container status \"a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184\": rpc error: code = NotFound desc = could not find container \"a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184\": container with ID starting with a622521ad664155c3a56b5e8f31d38b18624d0134ab681955ba146a1c4a45184 not found: ID does not exist"
Sep 09 20:55:31.846814 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.846758    2563 scope.go:117] "RemoveContainer" containerID="80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554"
Sep 09 20:55:31.847028 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:55:31.847014    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554\": container with ID starting with 80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554 not found: ID does not exist" containerID="80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554"
Sep 09 20:55:31.847074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.847032    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554"} err="failed to get container status \"80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554\": rpc error: code = NotFound desc = could not find container \"80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554\": container with ID starting with 80eaa46e933c6e304e59e48596b27f51d9ca05a23f7f00a3aaa60ab8e18a5554 not found: ID does not exist"
Sep 09 20:55:31.847074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.847044    2563 scope.go:117] "RemoveContainer" containerID="0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53"
Sep 09 20:55:31.847299 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:55:31.847282    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53\": container with ID starting with 0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53 not found: ID does not exist" containerID="0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53"
Sep 09 20:55:31.847349 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.847304    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53"} err="failed to get container status \"0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53\": rpc error: code = NotFound desc = could not find container \"0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53\": container with ID starting with 0d28b9ab30d54db230e7e6df4d5f855867bd2389ebbc0fb0a1d04ae5da7b4b53 not found: ID does not exist"
Sep 09 20:55:31.848032 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.848013    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-vfzj7"]
Sep 09 20:55:31.910554 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.910524    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:55:31.910554 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.910555    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:55:31.910751 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.910570    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcfsx\" (UniqueName: \"kubernetes.io/projected/b03402cb-18a3-463f-a6c6-45b5cda892d5-kube-api-access-hcfsx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:55:31.910751 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.910584    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03402cb-18a3-463f-a6c6-45b5cda892d5-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:55:31.910751 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:31.910598    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b03402cb-18a3-463f-a6c6-45b5cda892d5-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:55:33.424611 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:33.424582    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" path="/var/lib/kubelet/pods/b03402cb-18a3-463f-a6c6-45b5cda892d5/volumes"
Sep 09 20:55:50.914598 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.914560    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"]
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915028    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="storage-initializer"
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915043    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="storage-initializer"
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915052    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kube-rbac-proxy"
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915058    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kube-rbac-proxy"
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915066    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="storage-initializer"
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915075    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="storage-initializer"
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915096    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container"
Sep 09 20:55:50.915101 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915102    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915110    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915115    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915123    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kube-rbac-proxy"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915128    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kube-rbac-proxy"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915186    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kube-rbac-proxy"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915194    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03402cb-18a3-463f-a6c6-45b5cda892d5" containerName="kserve-container"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915202    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kube-rbac-proxy"
Sep 09 20:55:50.915363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.915210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3660280-718c-4a95-bce8-3e8397496975" containerName="kserve-container"
Sep 09 20:55:50.918445 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.918427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:50.920946 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.920918    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 09 20:55:50.920946 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.920919    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 09 20:55:50.921140 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.920955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:55:50.924402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.924377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"]
Sep 09 20:55:50.964784 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.964751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:50.964944 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.964791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2b7d6b3-4e7d-43f3-a131-88f470d27102-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:50.964944 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.964812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:50.964944 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.964883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:50.964944 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:50.964914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7qj\" (UniqueName: \"kubernetes.io/projected/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-api-access-8c7qj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066136 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066136 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2b7d6b3-4e7d-43f3-a131-88f470d27102-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066360 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066360 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066360 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7qj\" (UniqueName: \"kubernetes.io/projected/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-api-access-8c7qj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066705 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066846 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.066924 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.066905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.068750 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.068732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2b7d6b3-4e7d-43f3-a131-88f470d27102-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.074069 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.074042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7qj\" (UniqueName: \"kubernetes.io/projected/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-api-access-8c7qj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.230876 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.230771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:51.505417 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.505393    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"]
Sep 09 20:55:51.507538 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:55:51.507508    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b7d6b3_4e7d_43f3_a131_88f470d27102.slice/crio-e042c18f0b29cd7ccfad6629484645402701bcb14a30224f4e12329c8919d308 WatchSource:0}: Error finding container e042c18f0b29cd7ccfad6629484645402701bcb14a30224f4e12329c8919d308: Status 404 returned error can't find the container with id e042c18f0b29cd7ccfad6629484645402701bcb14a30224f4e12329c8919d308
Sep 09 20:55:51.916484 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.916444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerStarted","Data":"b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc"}
Sep 09 20:55:51.916981 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:51.916521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerStarted","Data":"e042c18f0b29cd7ccfad6629484645402701bcb14a30224f4e12329c8919d308"}
Sep 09 20:55:53.929048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:53.929007    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerID="b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc" exitCode=0
Sep 09 20:55:53.929460 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:53.929055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerDied","Data":"b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc"}
Sep 09 20:55:54.936080 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:54.936046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerStarted","Data":"9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4"}
Sep 09 20:55:54.936430 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:54.936106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerStarted","Data":"82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29"}
Sep 09 20:55:54.936430 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:54.936294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:54.954174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:54.954123    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podStartSLOduration=3.538600102 podStartE2EDuration="4.954102575s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.415502473s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:54.953953898 +0000 UTC m=+2681.993123864" watchObservedRunningTime="2026-09-09 20:55:54.954102575 +0000 UTC m=+2681.993272528"
Sep 09 20:55:55.940230 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:55.940200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:55:55.941274 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:55.941250    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:55:56.945690 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:55:56.945629    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:56:01.950977 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:01.950947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:56:01.951495 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:01.951454    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:56:11.951414 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:11.951371    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:56:21.951749 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:21.951708    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:56:31.951378 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:31.951341    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:56:41.951663 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:41.951611    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:56:51.952332 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:51.952281    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:56:57.020721 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:57.020688    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"]
Sep 09 20:56:57.021123 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:57.020922    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" containerID="cri-o://ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182" gracePeriod=30
Sep 09 20:56:57.021123 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:57.020962    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kube-rbac-proxy" containerID="cri-o://7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086" gracePeriod=30
Sep 09 20:56:57.227602 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:57.227565    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerID="7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086" exitCode=2
Sep 09 20:56:57.227789 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:57.227637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerDied","Data":"7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086"}
Sep 09 20:56:59.408442 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:59.408404    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:56:59.412962 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:56:59.412938    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:57:01.952427 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:01.952393    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:57:01.972881 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:01.972856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:57:02.082811 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.082720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-proxy-tls\") pod \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") "
Sep 09 20:57:02.083013 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.082830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kserve-provision-location\") pod \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") "
Sep 09 20:57:02.083013 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.082868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmhfh\" (UniqueName: \"kubernetes.io/projected/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-api-access-zmhfh\") pod \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") "
Sep 09 20:57:02.083013 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.082919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-cabundle-cert\") pod \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") "
Sep 09 20:57:02.083013 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.082943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-rbac-proxy-sar-config\") pod \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\" (UID: \"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8\") "
Sep 09 20:57:02.083256 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.083181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" (UID: "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:02.083363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.083332    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.083597 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.083453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" (UID: "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:02.083774 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.083639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" (UID: "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:02.085569 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.085542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-api-access-zmhfh" (OuterVolumeSpecName: "kube-api-access-zmhfh") pod "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" (UID: "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8"). InnerVolumeSpecName "kube-api-access-zmhfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:02.085698 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.085609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" (UID: "fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:02.184491 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.184451    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhfh\" (UniqueName: \"kubernetes.io/projected/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-api-access-zmhfh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.184675 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.184495    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.184675 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.184513    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.184675 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.184528    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.230232 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230196    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"]
Sep 09 20:57:02.230557 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230545    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="storage-initializer"
Sep 09 20:57:02.230613 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230559    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="storage-initializer"
Sep 09 20:57:02.230613 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230579    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container"
Sep 09 20:57:02.230613 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container"
Sep 09 20:57:02.230613 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230596    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kube-rbac-proxy"
Sep 09 20:57:02.230613 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230602    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kube-rbac-proxy"
Sep 09 20:57:02.230800 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230674    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kserve-container"
Sep 09 20:57:02.230800 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.230686    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerName="kube-rbac-proxy"
Sep 09 20:57:02.233983 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.233955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.236953 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.236927    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 09 20:57:02.237083 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.236935    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:57:02.248348 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.248319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"]
Sep 09 20:57:02.253903 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.253875    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" containerID="ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182" exitCode=0
Sep 09 20:57:02.254060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.253914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerDied","Data":"ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182"}
Sep 09 20:57:02.254060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.253947    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"
Sep 09 20:57:02.254060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.253976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z" event={"ID":"fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8","Type":"ContainerDied","Data":"60774ee3200887302f61f14265a2863ca29aa6d44752cbb7d0f7c111bac4a565"}
Sep 09 20:57:02.254060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.254013    2563 scope.go:117] "RemoveContainer" containerID="7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086"
Sep 09 20:57:02.263511 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.263489    2563 scope.go:117] "RemoveContainer" containerID="ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182"
Sep 09 20:57:02.273047 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.273028    2563 scope.go:117] "RemoveContainer" containerID="79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908"
Sep 09 20:57:02.282171 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.282151    2563 scope.go:117] "RemoveContainer" containerID="7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086"
Sep 09 20:57:02.282500 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:57:02.282473    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086\": container with ID starting with 7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086 not found: ID does not exist" containerID="7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086"
Sep 09 20:57:02.282562 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.282510    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086"} err="failed to get container status \"7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086\": rpc error: code = NotFound desc = could not find container \"7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086\": container with ID starting with 7bce63aea8fc2b9b0fac1956eebf8321e003030b82e597c2006387162afad086 not found: ID does not exist"
Sep 09 20:57:02.282562 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.282532    2563 scope.go:117] "RemoveContainer" containerID="ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182"
Sep 09 20:57:02.282833 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:57:02.282816    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182\": container with ID starting with ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182 not found: ID does not exist" containerID="ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182"
Sep 09 20:57:02.282892 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.282851    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182"} err="failed to get container status \"ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182\": rpc error: code = NotFound desc = could not find container \"ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182\": container with ID starting with ac0853a56c605cc07977348c67fdefc3ed2c31108f88983abcdd9a3ba83eb182 not found: ID does not exist"
Sep 09 20:57:02.282892 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.282868    2563 scope.go:117] "RemoveContainer" containerID="79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908"
Sep 09 20:57:02.283117 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:57:02.283101    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908\": container with ID starting with 79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908 not found: ID does not exist" containerID="79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908"
Sep 09 20:57:02.283153 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.283121    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908"} err="failed to get container status \"79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908\": rpc error: code = NotFound desc = could not find container \"79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908\": container with ID starting with 79b6ac2c4c8efa8d87b7e5a0d1c9122a89ae0f7c797452e8cc176aad2977a908 not found: ID does not exist"
Sep 09 20:57:02.285576 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.285550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.285685 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.285623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7j88\" (UniqueName: \"kubernetes.io/projected/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-api-access-k7j88\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.285749 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.285730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e3f717-5583-4453-8efa-839dadae0cc0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.285786 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.285763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.285820 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.285808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e3f717-5583-4453-8efa-839dadae0cc0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.289745 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.289720    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"]
Sep 09 20:57:02.291477 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.291455    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-bht9z"]
Sep 09 20:57:02.334705 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.334603    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"]
Sep 09 20:57:02.339825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.339791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.342432 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.342405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:57:02.349844 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.349819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"]
Sep 09 20:57:02.386476 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.386666 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e3f717-5583-4453-8efa-839dadae0cc0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.386666 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.386666 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.386666 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e3f717-5583-4453-8efa-839dadae0cc0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.386666 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.386919 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.386919 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7gp\" (UniqueName: \"kubernetes.io/projected/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-api-access-hb7gp\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.386919 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.386919 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.386799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7j88\" (UniqueName: \"kubernetes.io/projected/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-api-access-k7j88\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.387074 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.387008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e3f717-5583-4453-8efa-839dadae0cc0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.387488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.387376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.387488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.387413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.389439 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.389417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e3f717-5583-4453-8efa-839dadae0cc0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.395277 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.395247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7j88\" (UniqueName: \"kubernetes.io/projected/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-api-access-k7j88\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.487490 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.487452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.487692 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.487526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.487692 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.487556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.487692 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.487604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7gp\" (UniqueName: \"kubernetes.io/projected/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-api-access-hb7gp\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.487862 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.487693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.488066 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.488040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.488299 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.488276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.488350 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.488276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.490221 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.490198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.496993 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.496971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7gp\" (UniqueName: \"kubernetes.io/projected/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-api-access-hb7gp\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-xnbql\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.544612 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.544569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:02.652279 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.652225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:02.824203 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.824174    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"]
Sep 09 20:57:02.826402 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:57:02.826368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e3f717_5583_4453_8efa_839dadae0cc0.slice/crio-d7a13a18f635fb2a7587889eb434836476ecf52f4ba45c8f461ceeee395f9877 WatchSource:0}: Error finding container d7a13a18f635fb2a7587889eb434836476ecf52f4ba45c8f461ceeee395f9877: Status 404 returned error can't find the container with id d7a13a18f635fb2a7587889eb434836476ecf52f4ba45c8f461ceeee395f9877
Sep 09 20:57:02.928596 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:02.928571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"]
Sep 09 20:57:02.930777 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:57:02.930748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb644c8ac_b2c9_4ef2_b447_c7af76ca2b59.slice/crio-628f1037085b866efb296061238dc294544d7996ea03fe755366ebf407fc09c6 WatchSource:0}: Error finding container 628f1037085b866efb296061238dc294544d7996ea03fe755366ebf407fc09c6: Status 404 returned error can't find the container with id 628f1037085b866efb296061238dc294544d7996ea03fe755366ebf407fc09c6
Sep 09 20:57:03.271996 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:03.271881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerStarted","Data":"adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7"}
Sep 09 20:57:03.272433 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:03.271999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerStarted","Data":"628f1037085b866efb296061238dc294544d7996ea03fe755366ebf407fc09c6"}
Sep 09 20:57:03.275524 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:03.275489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerStarted","Data":"9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b"}
Sep 09 20:57:03.275723 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:03.275552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerStarted","Data":"d7a13a18f635fb2a7587889eb434836476ecf52f4ba45c8f461ceeee395f9877"}
Sep 09 20:57:03.425547 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:03.425513    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8" path="/var/lib/kubelet/pods/fdc9de2f-0502-4f34-bb13-d2d6f16ed3f8/volumes"
Sep 09 20:57:07.297180 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:07.297147    2563 generic.go:356] "Generic (PLEG): container finished" podID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerID="adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7" exitCode=0
Sep 09 20:57:07.297709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:07.297233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerDied","Data":"adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7"}
Sep 09 20:57:07.298873 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:07.298852    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerID="9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b" exitCode=0
Sep 09 20:57:07.298952 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:07.298933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerDied","Data":"9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b"}
Sep 09 20:57:08.311627 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.311588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerStarted","Data":"cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548"}
Sep 09 20:57:08.312093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.311702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerStarted","Data":"51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd"}
Sep 09 20:57:08.312093 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.312060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:08.312252 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.312185    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:08.313728 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.313698    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:57:08.314037 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.314017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerStarted","Data":"0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567"}
Sep 09 20:57:08.314126 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.314072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerStarted","Data":"33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7"}
Sep 09 20:57:08.314343 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.314326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:08.314420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.314376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:08.315445 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.315422    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:57:08.333905 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.333866    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podStartSLOduration=2.540536215 podStartE2EDuration="6.333843064s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793306849s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:08.33153805 +0000 UTC m=+2755.370708001" watchObservedRunningTime="2026-09-09 20:57:08.333843064 +0000 UTC m=+2755.373013014"
Sep 09 20:57:08.353711 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:08.353642    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podStartSLOduration=2.561457915 podStartE2EDuration="6.353625219s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792167304s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:08.35199487 +0000 UTC m=+2755.391164834" watchObservedRunningTime="2026-09-09 20:57:08.353625219 +0000 UTC m=+2755.392795171"
Sep 09 20:57:09.319852 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:09.319808    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:57:09.320219 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:09.319909    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:57:11.952596 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:11.952563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:57:14.325716 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:14.325685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:57:14.326119 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:14.325773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:57:14.326280 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:14.326259    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:57:14.326360 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:14.326337    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:57:20.940344 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:20.940307    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"]
Sep 09 20:57:20.940853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:20.940617    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" containerID="cri-o://82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29" gracePeriod=30
Sep 09 20:57:20.940853 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:20.940666    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kube-rbac-proxy" containerID="cri-o://9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4" gracePeriod=30
Sep 09 20:57:21.375404 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:21.375370    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerID="9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4" exitCode=2
Sep 09 20:57:21.375552 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:21.375430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerDied","Data":"9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4"}
Sep 09 20:57:21.946701 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:21.946635    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 20:57:21.952048 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:21.952003    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:57:24.327211 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:24.327168    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:57:24.327596 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:24.327169    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:57:26.186164 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.186131    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:57:26.307971 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.307946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kserve-provision-location\") pod \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") "
Sep 09 20:57:26.308154 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.308026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c7qj\" (UniqueName: \"kubernetes.io/projected/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-api-access-8c7qj\") pod \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") "
Sep 09 20:57:26.308154 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.308055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2b7d6b3-4e7d-43f3-a131-88f470d27102-proxy-tls\") pod \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") "
Sep 09 20:57:26.308154 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.308085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-cabundle-cert\") pod \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") "
Sep 09 20:57:26.308154 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.308099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-rbac-proxy-sar-config\") pod \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\" (UID: \"a2b7d6b3-4e7d-43f3-a131-88f470d27102\") "
Sep 09 20:57:26.308374 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.308272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2b7d6b3-4e7d-43f3-a131-88f470d27102" (UID: "a2b7d6b3-4e7d-43f3-a131-88f470d27102"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:26.308556 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.308527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2b7d6b3-4e7d-43f3-a131-88f470d27102" (UID: "a2b7d6b3-4e7d-43f3-a131-88f470d27102"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:26.308556 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.308533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2b7d6b3-4e7d-43f3-a131-88f470d27102" (UID: "a2b7d6b3-4e7d-43f3-a131-88f470d27102"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:26.310318 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.310288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-api-access-8c7qj" (OuterVolumeSpecName: "kube-api-access-8c7qj") pod "a2b7d6b3-4e7d-43f3-a131-88f470d27102" (UID: "a2b7d6b3-4e7d-43f3-a131-88f470d27102"). InnerVolumeSpecName "kube-api-access-8c7qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:26.310404 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.310376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b7d6b3-4e7d-43f3-a131-88f470d27102-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2b7d6b3-4e7d-43f3-a131-88f470d27102" (UID: "a2b7d6b3-4e7d-43f3-a131-88f470d27102"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:26.400776 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.400745    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerID="82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29" exitCode=0
Sep 09 20:57:26.400953 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.400803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"
Sep 09 20:57:26.400953 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.400818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerDied","Data":"82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29"}
Sep 09 20:57:26.400953 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.400867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62" event={"ID":"a2b7d6b3-4e7d-43f3-a131-88f470d27102","Type":"ContainerDied","Data":"e042c18f0b29cd7ccfad6629484645402701bcb14a30224f4e12329c8919d308"}
Sep 09 20:57:26.400953 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.400891    2563 scope.go:117] "RemoveContainer" containerID="9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4"
Sep 09 20:57:26.408881 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.408853    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:26.409001 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.408885    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7qj\" (UniqueName: \"kubernetes.io/projected/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-api-access-8c7qj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:26.409001 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.408901    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2b7d6b3-4e7d-43f3-a131-88f470d27102-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:26.409001 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.408915    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:26.409001 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.408928    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2b7d6b3-4e7d-43f3-a131-88f470d27102-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:57:26.410366 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.410347    2563 scope.go:117] "RemoveContainer" containerID="82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29"
Sep 09 20:57:26.418047 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.418030    2563 scope.go:117] "RemoveContainer" containerID="b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc"
Sep 09 20:57:26.426402 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.426383    2563 scope.go:117] "RemoveContainer" containerID="9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4"
Sep 09 20:57:26.426726 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:57:26.426702    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4\": container with ID starting with 9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4 not found: ID does not exist" containerID="9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4"
Sep 09 20:57:26.426774 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.426741    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4"} err="failed to get container status \"9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4\": rpc error: code = NotFound desc = could not find container \"9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4\": container with ID starting with 9f31f8828035ac7ca789db99d123f370e3a56dff6792a659204bce256a9ba7b4 not found: ID does not exist"
Sep 09 20:57:26.426774 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.426764    2563 scope.go:117] "RemoveContainer" containerID="82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29"
Sep 09 20:57:26.427023 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:57:26.427006    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29\": container with ID starting with 82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29 not found: ID does not exist" containerID="82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29"
Sep 09 20:57:26.427067 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.427030    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29"} err="failed to get container status \"82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29\": rpc error: code = NotFound desc = could not find container \"82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29\": container with ID starting with 82402fb44980db9cef97b1292e74e8b56dff5f88eb93e7a07be09ac95b537b29 not found: ID does not exist"
Sep 09 20:57:26.427067 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.427044    2563 scope.go:117] "RemoveContainer" containerID="b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc"
Sep 09 20:57:26.427269 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:57:26.427250    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc\": container with ID starting with b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc not found: ID does not exist" containerID="b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc"
Sep 09 20:57:26.427312 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.427272    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc"} err="failed to get container status \"b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc\": rpc error: code = NotFound desc = could not find container \"b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc\": container with ID starting with b51d4fa0ef208b12f6c0be3e2c9460743917fc47817628c7032480ee7e0b37bc not found: ID does not exist"
Sep 09 20:57:26.435545 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.435517    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"]
Sep 09 20:57:26.446558 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:26.446534    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-747f76cf77-5wc62"]
Sep 09 20:57:27.425252 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:27.425210    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" path="/var/lib/kubelet/pods/a2b7d6b3-4e7d-43f3-a131-88f470d27102/volumes"
Sep 09 20:57:34.327060 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:34.327021    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:57:34.327470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:34.327024    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:57:44.326800 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:44.326757    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:57:44.327208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:44.326757    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:57:54.327126 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:54.327084    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:57:54.327696 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:57:54.327084    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:58:04.326926 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:04.326886    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:58:04.327397 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:04.326890    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:58:14.326805 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:14.326765    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:58:14.327197 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:14.326766    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:58:21.419338 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:21.419292    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:58:24.327956 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:24.327927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:58:31.424340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:31.424309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:58:32.260699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.260665    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"]
Sep 09 20:58:32.260997 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.260946    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container" containerID="cri-o://33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7" gracePeriod=30
Sep 09 20:58:32.261112 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.261020    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kube-rbac-proxy" containerID="cri-o://0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567" gracePeriod=30
Sep 09 20:58:32.293717 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.293625    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"]
Sep 09 20:58:32.294177 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.294119    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" containerID="cri-o://51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd" gracePeriod=30
Sep 09 20:58:32.294312 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.294209    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kube-rbac-proxy" containerID="cri-o://cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548" gracePeriod=30
Sep 09 20:58:32.707781 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.707748    2563 generic.go:356] "Generic (PLEG): container finished" podID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerID="cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548" exitCode=2
Sep 09 20:58:32.708260 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.707821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerDied","Data":"cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548"}
Sep 09 20:58:32.712723 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.712698    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerID="0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567" exitCode=2
Sep 09 20:58:32.712860 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:32.712771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerDied","Data":"0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567"}
Sep 09 20:58:34.320355 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:34.320309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 20:58:34.320767 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:34.320327    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 09 20:58:34.326351 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:34.326322    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:58:37.354537 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.354506    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:58:37.413549 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.413523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-rbac-proxy-sar-config\") pod \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") "
Sep 09 20:58:37.413720 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.413564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-proxy-tls\") pod \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") "
Sep 09 20:58:37.413720 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.413579    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-cabundle-cert\") pod \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") "
Sep 09 20:58:37.413720 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.413671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb7gp\" (UniqueName: \"kubernetes.io/projected/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-api-access-hb7gp\") pod \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") "
Sep 09 20:58:37.413907 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.413724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kserve-provision-location\") pod \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\" (UID: \"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59\") "
Sep 09 20:58:37.414037 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.414018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" (UID: "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:37.414113 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.414030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" (UID: "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:37.414113 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.414042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" (UID: "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:37.414214 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.414197    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.414258 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.414216    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.414258 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.414232    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.416100 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.416078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" (UID: "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:37.416208 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.416111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-api-access-hb7gp" (OuterVolumeSpecName: "kube-api-access-hb7gp") pod "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" (UID: "b644c8ac-b2c9-4ef2-b447-c7af76ca2b59"). InnerVolumeSpecName "kube-api-access-hb7gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:37.417320 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.417302    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:58:37.514566 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514533    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7j88\" (UniqueName: \"kubernetes.io/projected/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-api-access-k7j88\") pod \"f2e3f717-5583-4453-8efa-839dadae0cc0\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") "
Sep 09 20:58:37.514746 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-rbac-proxy-sar-config\") pod \"f2e3f717-5583-4453-8efa-839dadae0cc0\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") "
Sep 09 20:58:37.514746 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e3f717-5583-4453-8efa-839dadae0cc0-kserve-provision-location\") pod \"f2e3f717-5583-4453-8efa-839dadae0cc0\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") "
Sep 09 20:58:37.514746 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-cabundle-cert\") pod \"f2e3f717-5583-4453-8efa-839dadae0cc0\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") "
Sep 09 20:58:37.514746 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e3f717-5583-4453-8efa-839dadae0cc0-proxy-tls\") pod \"f2e3f717-5583-4453-8efa-839dadae0cc0\" (UID: \"f2e3f717-5583-4453-8efa-839dadae0cc0\") "
Sep 09 20:58:37.515108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514923    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.515108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514942    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb7gp\" (UniqueName: \"kubernetes.io/projected/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59-kube-api-access-hb7gp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.515108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.514996    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e3f717-5583-4453-8efa-839dadae0cc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2e3f717-5583-4453-8efa-839dadae0cc0" (UID: "f2e3f717-5583-4453-8efa-839dadae0cc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:37.515108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.515011    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2e3f717-5583-4453-8efa-839dadae0cc0" (UID: "f2e3f717-5583-4453-8efa-839dadae0cc0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:37.515108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.515095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2e3f717-5583-4453-8efa-839dadae0cc0" (UID: "f2e3f717-5583-4453-8efa-839dadae0cc0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:37.516842 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.516822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-api-access-k7j88" (OuterVolumeSpecName: "kube-api-access-k7j88") pod "f2e3f717-5583-4453-8efa-839dadae0cc0" (UID: "f2e3f717-5583-4453-8efa-839dadae0cc0"). InnerVolumeSpecName "kube-api-access-k7j88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:37.516906 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.516871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e3f717-5583-4453-8efa-839dadae0cc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2e3f717-5583-4453-8efa-839dadae0cc0" (UID: "f2e3f717-5583-4453-8efa-839dadae0cc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:37.616151 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.616118    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7j88\" (UniqueName: \"kubernetes.io/projected/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-api-access-k7j88\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.616151 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.616145    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.616151 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.616155    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e3f717-5583-4453-8efa-839dadae0cc0-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.616362 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.616163    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e3f717-5583-4453-8efa-839dadae0cc0-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.616362 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.616172    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e3f717-5583-4453-8efa-839dadae0cc0-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.736900 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.736816    2563 generic.go:356] "Generic (PLEG): container finished" podID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerID="51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd" exitCode=0
Sep 09 20:58:37.736900 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.736852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerDied","Data":"51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd"}
Sep 09 20:58:37.736900 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.736873    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"
Sep 09 20:58:37.737105 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.736912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql" event={"ID":"b644c8ac-b2c9-4ef2-b447-c7af76ca2b59","Type":"ContainerDied","Data":"628f1037085b866efb296061238dc294544d7996ea03fe755366ebf407fc09c6"}
Sep 09 20:58:37.737105 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.736939    2563 scope.go:117] "RemoveContainer" containerID="cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548"
Sep 09 20:58:37.739233 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.739143    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerID="33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7" exitCode=0
Sep 09 20:58:37.739233 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.739177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerDied","Data":"33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7"}
Sep 09 20:58:37.739233 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.739220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"
Sep 09 20:58:37.739470 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.739265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq" event={"ID":"f2e3f717-5583-4453-8efa-839dadae0cc0","Type":"ContainerDied","Data":"d7a13a18f635fb2a7587889eb434836476ecf52f4ba45c8f461ceeee395f9877"}
Sep 09 20:58:37.745823 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.745803    2563 scope.go:117] "RemoveContainer" containerID="51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd"
Sep 09 20:58:37.753825 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.753808    2563 scope.go:117] "RemoveContainer" containerID="adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7"
Sep 09 20:58:37.759599 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.759577    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"]
Sep 09 20:58:37.761597 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.761556    2563 scope.go:117] "RemoveContainer" containerID="cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548"
Sep 09 20:58:37.762227 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:58:37.762207    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548\": container with ID starting with cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548 not found: ID does not exist" containerID="cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548"
Sep 09 20:58:37.762314 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.762235    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548"} err="failed to get container status \"cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548\": rpc error: code = NotFound desc = could not find container \"cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548\": container with ID starting with cdf46fcbc76a20186ef5989c63c489410e9c09ffcd4d01abb3c2a642264f4548 not found: ID does not exist"
Sep 09 20:58:37.762314 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.762261    2563 scope.go:117] "RemoveContainer" containerID="51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd"
Sep 09 20:58:37.762582 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:58:37.762560    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd\": container with ID starting with 51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd not found: ID does not exist" containerID="51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd"
Sep 09 20:58:37.762634 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.762585    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd"} err="failed to get container status \"51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd\": rpc error: code = NotFound desc = could not find container \"51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd\": container with ID starting with 51eb97bf2c2f0809ab96c8ff432eed2fd6cb5adbc6e7a146651be048dfe5f6bd not found: ID does not exist"
Sep 09 20:58:37.762634 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.762598    2563 scope.go:117] "RemoveContainer" containerID="adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7"
Sep 09 20:58:37.763152 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:58:37.763133    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7\": container with ID starting with adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7 not found: ID does not exist" containerID="adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7"
Sep 09 20:58:37.763209 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.763159    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7"} err="failed to get container status \"adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7\": rpc error: code = NotFound desc = could not find container \"adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7\": container with ID starting with adebb7e275728d62d93ed31484b845c803338bd35fb18e74fcd9b159074c5aa7 not found: ID does not exist"
Sep 09 20:58:37.763209 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.763175    2563 scope.go:117] "RemoveContainer" containerID="0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567"
Sep 09 20:58:37.763968 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.763953    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-xnbql"]
Sep 09 20:58:37.770568 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.770552    2563 scope.go:117] "RemoveContainer" containerID="33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7"
Sep 09 20:58:37.778488 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.778467    2563 scope.go:117] "RemoveContainer" containerID="9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b"
Sep 09 20:58:37.779751 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.779732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"]
Sep 09 20:58:37.783958 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.783932    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-4mlrq"]
Sep 09 20:58:37.786961 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.786946    2563 scope.go:117] "RemoveContainer" containerID="0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567"
Sep 09 20:58:37.787247 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:58:37.787229    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567\": container with ID starting with 0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567 not found: ID does not exist" containerID="0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567"
Sep 09 20:58:37.787296 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.787255    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567"} err="failed to get container status \"0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567\": rpc error: code = NotFound desc = could not find container \"0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567\": container with ID starting with 0c09ca4a5d728abd9682e12914fd3694d40f26ba61e565286eb80810a5f30567 not found: ID does not exist"
Sep 09 20:58:37.787296 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.787275    2563 scope.go:117] "RemoveContainer" containerID="33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7"
Sep 09 20:58:37.787540 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:58:37.787521    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7\": container with ID starting with 33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7 not found: ID does not exist" containerID="33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7"
Sep 09 20:58:37.787582 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.787547    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7"} err="failed to get container status \"33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7\": rpc error: code = NotFound desc = could not find container \"33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7\": container with ID starting with 33f76af06420a828c6cb607e572f0e4a0bb1556859bf4f51ebf8d660fb2abfc7 not found: ID does not exist"
Sep 09 20:58:37.787582 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.787564    2563 scope.go:117] "RemoveContainer" containerID="9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b"
Sep 09 20:58:37.787848 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:58:37.787834    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b\": container with ID starting with 9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b not found: ID does not exist" containerID="9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b"
Sep 09 20:58:37.787913 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:37.787851    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b"} err="failed to get container status \"9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b\": rpc error: code = NotFound desc = could not find container \"9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b\": container with ID starting with 9f1c4e21e8a739feacbc5dd78df6a9b45cface0f59452e0c66d23d913d2ed93b not found: ID does not exist"
Sep 09 20:58:39.424932 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:39.424893    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" path="/var/lib/kubelet/pods/b644c8ac-b2c9-4ef2-b447-c7af76ca2b59/volumes"
Sep 09 20:58:39.425444 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:39.425429    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" path="/var/lib/kubelet/pods/f2e3f717-5583-4453-8efa-839dadae0cc0/volumes"
Sep 09 20:58:42.556539 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"]
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556848    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="storage-initializer"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556860    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="storage-initializer"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556870    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kube-rbac-proxy"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556876    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kube-rbac-proxy"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556884    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kube-rbac-proxy"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556891    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kube-rbac-proxy"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556896    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556901    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556912    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556917    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="storage-initializer"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556930    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="storage-initializer"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556935    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kube-rbac-proxy"
Sep 09 20:58:42.556937 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556940    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kube-rbac-proxy"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556947    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556953    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556960    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="storage-initializer"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.556965    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="storage-initializer"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.557017    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kserve-container"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.557026    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kube-rbac-proxy"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.557033    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e3f717-5583-4453-8efa-839dadae0cc0" containerName="kserve-container"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.557039    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kube-rbac-proxy"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.557045    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b644c8ac-b2c9-4ef2-b447-c7af76ca2b59" containerName="kserve-container"
Sep 09 20:58:42.557358 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.557051    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b7d6b3-4e7d-43f3-a131-88f470d27102" containerName="kube-rbac-proxy"
Sep 09 20:58:42.561200 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.561176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.564091 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.564017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 09 20:58:42.564091 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.564071    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:58:42.564354 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.564102    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:58:42.564354 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.564079    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:58:42.564354 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.564238    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 20:58:42.564563 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.564435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 09 20:58:42.565363 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.565345    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:58:42.572173 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.572150    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"]
Sep 09 20:58:42.659051 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.659018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.659272 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.659069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.659272 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.659120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfmft\" (UniqueName: \"kubernetes.io/projected/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-api-access-mfmft\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.659272 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.659149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.659272 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.659165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.760174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.760174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.760420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.760420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.760420 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfmft\" (UniqueName: \"kubernetes.io/projected/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-api-access-mfmft\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.760784 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.760947 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.761004 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.760933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.762831 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.762814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.770250 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.770228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfmft\" (UniqueName: \"kubernetes.io/projected/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-api-access-mfmft\") pod \"isvc-lightgbm-predictor-54f85ddf88-qtthv\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:42.873395 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:42.873303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:43.148498 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:43.148418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"]
Sep 09 20:58:43.151493 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:58:43.151465    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b8a4c8_f6fe_42d0_a715_64f35ea72d72.slice/crio-1aa75dc79d3263f76c024ec7bf1bca03751d269a79a77f465c3208abf1e73139 WatchSource:0}: Error finding container 1aa75dc79d3263f76c024ec7bf1bca03751d269a79a77f465c3208abf1e73139: Status 404 returned error can't find the container with id 1aa75dc79d3263f76c024ec7bf1bca03751d269a79a77f465c3208abf1e73139
Sep 09 20:58:43.774875 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:43.774829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerStarted","Data":"526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142"}
Sep 09 20:58:43.775340 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:43.774901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerStarted","Data":"1aa75dc79d3263f76c024ec7bf1bca03751d269a79a77f465c3208abf1e73139"}
Sep 09 20:58:47.794258 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:47.794224    2563 generic.go:356] "Generic (PLEG): container finished" podID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerID="526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142" exitCode=0
Sep 09 20:58:47.794643 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:47.794285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerDied","Data":"526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142"}
Sep 09 20:58:47.795707 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:47.795691    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:58:54.833294 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:54.833256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerStarted","Data":"dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1"}
Sep 09 20:58:54.833733 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:54.833325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerStarted","Data":"492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7"}
Sep 09 20:58:54.833733 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:54.833533    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:54.855574 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:54.855524    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podStartSLOduration=2.131511572 podStartE2EDuration="12.855508778s" totalImagesPullingTime="6.744451162s" totalInitContainerRuntime="3.979546044s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:54.853331401 +0000 UTC m=+2861.892501355" watchObservedRunningTime="2026-09-09 20:58:54.855508778 +0000 UTC m=+2861.894678729"
Sep 09 20:58:55.837943 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:55.837902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:58:55.839381 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:55.839349    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:58:56.842667 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:58:56.842619    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:59:01.847747 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:01.847709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 20:59:01.848387 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:01.848358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:59:04.358174 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.358132    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"]
Sep 09 20:59:04.361914 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.361891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.364792 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.364770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 09 20:59:04.364894 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.364874    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:59:04.365103 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.365088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 09 20:59:04.375846 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.375822    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"]
Sep 09 20:59:04.454226 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.454136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.454226 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.454192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.454440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.454277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e003bb-ed97-4fb5-ad18-831e30473b91-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.454440 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.454353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfg9s\" (UniqueName: \"kubernetes.io/projected/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-api-access-lfg9s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.555532 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.555483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfg9s\" (UniqueName: \"kubernetes.io/projected/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-api-access-lfg9s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.555765 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.555554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.555765 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.555600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.555765 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.555694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e003bb-ed97-4fb5-ad18-831e30473b91-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.555944 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:59:04.555780    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 09 20:59:04.555944 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:59:04.555883    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls podName:39e003bb-ed97-4fb5-ad18-831e30473b91 nodeName:}" failed. No retries permitted until 2026-09-09 20:59:05.055861186 +0000 UTC m=+2872.095031115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" (UID: "39e003bb-ed97-4fb5-ad18-831e30473b91") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 09 20:59:04.556118 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.556097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e003bb-ed97-4fb5-ad18-831e30473b91-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.556272 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.556253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:04.565686 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:04.565658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfg9s\" (UniqueName: \"kubernetes.io/projected/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-api-access-lfg9s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:05.060566 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:05.060522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:05.063108 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:05.063082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:05.272351 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:05.272312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:05.547534 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:05.547503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"]
Sep 09 20:59:05.551888 ip-10-0-129-130 kubenswrapper[2563]: W0909 20:59:05.551857    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e003bb_ed97_4fb5_ad18_831e30473b91.slice/crio-c9ac807223843bb74c354f0bc8d005737c7fa8060073a5b75aab290d6be925e1 WatchSource:0}: Error finding container c9ac807223843bb74c354f0bc8d005737c7fa8060073a5b75aab290d6be925e1: Status 404 returned error can't find the container with id c9ac807223843bb74c354f0bc8d005737c7fa8060073a5b75aab290d6be925e1
Sep 09 20:59:05.886328 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:05.886244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" event={"ID":"39e003bb-ed97-4fb5-ad18-831e30473b91","Type":"ContainerStarted","Data":"5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3"}
Sep 09 20:59:05.886328 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:05.886307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" event={"ID":"39e003bb-ed97-4fb5-ad18-831e30473b91","Type":"ContainerStarted","Data":"c9ac807223843bb74c354f0bc8d005737c7fa8060073a5b75aab290d6be925e1"}
Sep 09 20:59:11.848604 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:11.848561    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:59:12.920613 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:12.920579    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r_39e003bb-ed97-4fb5-ad18-831e30473b91/storage-initializer/0.log"
Sep 09 20:59:12.921018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:12.920622    2563 generic.go:356] "Generic (PLEG): container finished" podID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerID="5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3" exitCode=1
Sep 09 20:59:12.921018 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:12.920703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" event={"ID":"39e003bb-ed97-4fb5-ad18-831e30473b91","Type":"ContainerDied","Data":"5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3"}
Sep 09 20:59:13.927586 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:13.927553    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r_39e003bb-ed97-4fb5-ad18-831e30473b91/storage-initializer/0.log"
Sep 09 20:59:13.928009 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:13.927680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" event={"ID":"39e003bb-ed97-4fb5-ad18-831e30473b91","Type":"ContainerStarted","Data":"b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea"}
Sep 09 20:59:14.323291 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:14.323260    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"]
Sep 09 20:59:14.932056 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:14.932021    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerName="storage-initializer" containerID="cri-o://b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea" gracePeriod=30
Sep 09 20:59:18.586144 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.586118    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r_39e003bb-ed97-4fb5-ad18-831e30473b91/storage-initializer/1.log"
Sep 09 20:59:18.586553 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.586539    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r_39e003bb-ed97-4fb5-ad18-831e30473b91/storage-initializer/0.log"
Sep 09 20:59:18.587978 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.587962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:18.674022 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.673912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e003bb-ed97-4fb5-ad18-831e30473b91-kserve-provision-location\") pod \"39e003bb-ed97-4fb5-ad18-831e30473b91\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") "
Sep 09 20:59:18.674022 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.673983    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-rbac-proxy-sar-config\") pod \"39e003bb-ed97-4fb5-ad18-831e30473b91\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") "
Sep 09 20:59:18.674022 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.674014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls\") pod \"39e003bb-ed97-4fb5-ad18-831e30473b91\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") "
Sep 09 20:59:18.674294 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.674040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfg9s\" (UniqueName: \"kubernetes.io/projected/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-api-access-lfg9s\") pod \"39e003bb-ed97-4fb5-ad18-831e30473b91\" (UID: \"39e003bb-ed97-4fb5-ad18-831e30473b91\") "
Sep 09 20:59:18.674294 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.674221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e003bb-ed97-4fb5-ad18-831e30473b91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39e003bb-ed97-4fb5-ad18-831e30473b91" (UID: "39e003bb-ed97-4fb5-ad18-831e30473b91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:18.674380 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.674365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39e003bb-ed97-4fb5-ad18-831e30473b91" (UID: "39e003bb-ed97-4fb5-ad18-831e30473b91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:18.676266 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.676240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39e003bb-ed97-4fb5-ad18-831e30473b91" (UID: "39e003bb-ed97-4fb5-ad18-831e30473b91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:18.676396 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.676282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-api-access-lfg9s" (OuterVolumeSpecName: "kube-api-access-lfg9s") pod "39e003bb-ed97-4fb5-ad18-831e30473b91" (UID: "39e003bb-ed97-4fb5-ad18-831e30473b91"). InnerVolumeSpecName "kube-api-access-lfg9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:18.774628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.774590    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e003bb-ed97-4fb5-ad18-831e30473b91-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:59:18.774628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.774621    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:59:18.774628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.774631    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e003bb-ed97-4fb5-ad18-831e30473b91-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:59:18.774628 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.774639    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfg9s\" (UniqueName: \"kubernetes.io/projected/39e003bb-ed97-4fb5-ad18-831e30473b91-kube-api-access-lfg9s\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 20:59:18.952168 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.952079    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r_39e003bb-ed97-4fb5-ad18-831e30473b91/storage-initializer/1.log"
Sep 09 20:59:18.952519 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.952504    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r_39e003bb-ed97-4fb5-ad18-831e30473b91/storage-initializer/0.log"
Sep 09 20:59:18.952576 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.952539    2563 generic.go:356] "Generic (PLEG): container finished" podID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerID="b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea" exitCode=1
Sep 09 20:59:18.952610 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.952590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" event={"ID":"39e003bb-ed97-4fb5-ad18-831e30473b91","Type":"ContainerDied","Data":"b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea"}
Sep 09 20:59:18.952610 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.952597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"
Sep 09 20:59:18.952709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.952673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r" event={"ID":"39e003bb-ed97-4fb5-ad18-831e30473b91","Type":"ContainerDied","Data":"c9ac807223843bb74c354f0bc8d005737c7fa8060073a5b75aab290d6be925e1"}
Sep 09 20:59:18.952709 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.952702    2563 scope.go:117] "RemoveContainer" containerID="b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea"
Sep 09 20:59:18.961908 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.961887    2563 scope.go:117] "RemoveContainer" containerID="5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3"
Sep 09 20:59:18.969841 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.969822    2563 scope.go:117] "RemoveContainer" containerID="b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea"
Sep 09 20:59:18.970121 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:59:18.970097    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea\": container with ID starting with b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea not found: ID does not exist" containerID="b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea"
Sep 09 20:59:18.970192 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.970127    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea"} err="failed to get container status \"b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea\": rpc error: code = NotFound desc = could not find container \"b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea\": container with ID starting with b28f8688bc7f2e6c7e6722e9a6ff6498e8754936ba0738addc53bdd56aa82dea not found: ID does not exist"
Sep 09 20:59:18.970192 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.970143    2563 scope.go:117] "RemoveContainer" containerID="5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3"
Sep 09 20:59:18.970413 ip-10-0-129-130 kubenswrapper[2563]: E0909 20:59:18.970392    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3\": container with ID starting with 5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3 not found: ID does not exist" containerID="5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3"
Sep 09 20:59:18.970455 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:18.970420    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3"} err="failed to get container status \"5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3\": rpc error: code = NotFound desc = could not find container \"5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3\": container with ID starting with 5834d779ea603b75fa30dd86bedb3a6f3a598734db98c9a148150e23ecbf40e3 not found: ID does not exist"
Sep 09 20:59:19.013237 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:19.013193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"]
Sep 09 20:59:19.016604 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:19.016576    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-54bc7d8f-kcz8r"]
Sep 09 20:59:19.424955 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:19.424916    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" path="/var/lib/kubelet/pods/39e003bb-ed97-4fb5-ad18-831e30473b91/volumes"
Sep 09 20:59:21.848699 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:21.848634    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:59:31.848394 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:31.848353    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:59:41.849134 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:41.849094    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:59:51.848782 ip-10-0-129-130 kubenswrapper[2563]: I0909 20:59:51.848738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 21:00:01.849054 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:01.849013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 21:00:11.849342 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:11.849312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 21:00:12.583733 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:12.583699    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"]
Sep 09 21:00:12.583953 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:12.583923    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container" containerID="cri-o://492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7" gracePeriod=30
Sep 09 21:00:12.584114 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:12.583959    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kube-rbac-proxy" containerID="cri-o://dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1" gracePeriod=30
Sep 09 21:00:13.207606 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:13.207573    2563 generic.go:356] "Generic (PLEG): container finished" podID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerID="dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1" exitCode=2
Sep 09 21:00:13.207606 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:13.207611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerDied","Data":"dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1"}
Sep 09 21:00:16.843296 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:16.843252    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 09 21:00:17.763534 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.763499    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"]
Sep 09 21:00:17.763889 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.763875    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerName="storage-initializer"
Sep 09 21:00:17.763941 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.763892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerName="storage-initializer"
Sep 09 21:00:17.763941 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.763910    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerName="storage-initializer"
Sep 09 21:00:17.763941 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.763915    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerName="storage-initializer"
Sep 09 21:00:17.764044 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.763974    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerName="storage-initializer"
Sep 09 21:00:17.764044 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.763984    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e003bb-ed97-4fb5-ad18-831e30473b91" containerName="storage-initializer"
Sep 09 21:00:17.768572 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.768551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.771588 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.771570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 09 21:00:17.772103 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.772084    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:00:17.778038 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.777989    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"]
Sep 09 21:00:17.866363 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.866332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.866718 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.866381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.866718 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.866412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.866718 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.866456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcpx\" (UniqueName: \"kubernetes.io/projected/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-api-access-wwcpx\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.866718 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.866501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.967235 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.967202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.967388 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.967244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.967388 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.967287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcpx\" (UniqueName: \"kubernetes.io/projected/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-api-access-wwcpx\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.967388 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.967328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.967552 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.967420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.967552 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:17.967529    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 09 21:00:17.967673 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:17.967621    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls podName:7c218eb5-221d-49fc-bbf5-aa3d90ccebff nodeName:}" failed. No retries permitted until 2026-09-09 21:00:18.467596897 +0000 UTC m=+2945.506766849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" (UID: "7c218eb5-221d-49fc-bbf5-aa3d90ccebff") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 09 21:00:17.967673 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.967634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.968098 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.968074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.968145 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.968081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:17.980540 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:17.980511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcpx\" (UniqueName: \"kubernetes.io/projected/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-api-access-wwcpx\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:18.027951 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.027887    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 21:00:18.169893 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.169861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-cabundle-cert\") pod \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") "
Sep 09 21:00:18.169893 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.169895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kserve-provision-location\") pod \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") "
Sep 09 21:00:18.170114 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.169918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-rbac-proxy-sar-config\") pod \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") "
Sep 09 21:00:18.170114 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.169962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-proxy-tls\") pod \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") "
Sep 09 21:00:18.170114 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.169991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfmft\" (UniqueName: \"kubernetes.io/projected/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-api-access-mfmft\") pod \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\" (UID: \"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72\") "
Sep 09 21:00:18.170276 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.170194    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" (UID: "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:18.170276 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.170263    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:18.170393 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.170265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" (UID: "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:18.170393 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.170278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" (UID: "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:18.172231 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.172208    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-api-access-mfmft" (OuterVolumeSpecName: "kube-api-access-mfmft") pod "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" (UID: "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72"). InnerVolumeSpecName "kube-api-access-mfmft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:18.172231 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.172220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" (UID: "d5b8a4c8-f6fe-42d0-a715-64f35ea72d72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:18.232298 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.232271    2563 generic.go:356] "Generic (PLEG): container finished" podID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerID="492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7" exitCode=0
Sep 09 21:00:18.232451 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.232330    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"
Sep 09 21:00:18.232451 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.232366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerDied","Data":"492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7"}
Sep 09 21:00:18.232451 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.232429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv" event={"ID":"d5b8a4c8-f6fe-42d0-a715-64f35ea72d72","Type":"ContainerDied","Data":"1aa75dc79d3263f76c024ec7bf1bca03751d269a79a77f465c3208abf1e73139"}
Sep 09 21:00:18.232561 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.232459    2563 scope.go:117] "RemoveContainer" containerID="dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1"
Sep 09 21:00:18.241125 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.241109    2563 scope.go:117] "RemoveContainer" containerID="492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7"
Sep 09 21:00:18.249146 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.249130    2563 scope.go:117] "RemoveContainer" containerID="526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142"
Sep 09 21:00:18.255427 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.255403    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"]
Sep 09 21:00:18.257930 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.257912    2563 scope.go:117] "RemoveContainer" containerID="dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1"
Sep 09 21:00:18.258209 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:18.258189    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1\": container with ID starting with dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1 not found: ID does not exist" containerID="dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1"
Sep 09 21:00:18.258279 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.258219    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1"} err="failed to get container status \"dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1\": rpc error: code = NotFound desc = could not find container \"dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1\": container with ID starting with dc073b32d8afc27eeddf10b2f5f3d4fcb120065782d73968ecbf88fbef9497e1 not found: ID does not exist"
Sep 09 21:00:18.258279 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.258236    2563 scope.go:117] "RemoveContainer" containerID="492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7"
Sep 09 21:00:18.258493 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:18.258475    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7\": container with ID starting with 492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7 not found: ID does not exist" containerID="492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7"
Sep 09 21:00:18.258552 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.258502    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7"} err="failed to get container status \"492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7\": rpc error: code = NotFound desc = could not find container \"492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7\": container with ID starting with 492a76f06c16108502b8c57aaddc1fe1c8fae5f19d710431293c987b987bafa7 not found: ID does not exist"
Sep 09 21:00:18.258552 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.258516    2563 scope.go:117] "RemoveContainer" containerID="526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142"
Sep 09 21:00:18.258841 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:18.258817    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142\": container with ID starting with 526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142 not found: ID does not exist" containerID="526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142"
Sep 09 21:00:18.258926 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.258858    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142"} err="failed to get container status \"526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142\": rpc error: code = NotFound desc = could not find container \"526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142\": container with ID starting with 526cc113b0f8d1b64af6a0bfe0c018036ad608aeafbf2a19ddbc126b8e22a142 not found: ID does not exist"
Sep 09 21:00:18.258926 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.258878    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qtthv"]
Sep 09 21:00:18.271291 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.271269    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:18.271291 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.271289    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:18.271385 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.271298    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:18.271385 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.271308    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfmft\" (UniqueName: \"kubernetes.io/projected/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72-kube-api-access-mfmft\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:18.473084 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.473050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:18.475699 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.475678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:18.679268 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.679219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:18.953273 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:18.953252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"]
Sep 09 21:00:18.955740 ip-10-0-129-130 kubenswrapper[2563]: W0909 21:00:18.955715    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c218eb5_221d_49fc_bbf5_aa3d90ccebff.slice/crio-942e37c7371ee0c3489f77e94ebe3d8aa04e2a59a1220e6446600e4e3b1a9940 WatchSource:0}: Error finding container 942e37c7371ee0c3489f77e94ebe3d8aa04e2a59a1220e6446600e4e3b1a9940: Status 404 returned error can't find the container with id 942e37c7371ee0c3489f77e94ebe3d8aa04e2a59a1220e6446600e4e3b1a9940
Sep 09 21:00:19.241725 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:19.241609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerStarted","Data":"140f07c56050c437725723d0b355dcec8a43797ca37342c8aa67d23d253e901f"}
Sep 09 21:00:19.241725 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:19.241695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerStarted","Data":"942e37c7371ee0c3489f77e94ebe3d8aa04e2a59a1220e6446600e4e3b1a9940"}
Sep 09 21:00:19.430231 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:19.426733    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" path="/var/lib/kubelet/pods/d5b8a4c8-f6fe-42d0-a715-64f35ea72d72/volumes"
Sep 09 21:00:23.260835 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:23.260799    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerID="140f07c56050c437725723d0b355dcec8a43797ca37342c8aa67d23d253e901f" exitCode=0
Sep 09 21:00:23.260835 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:23.260838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerDied","Data":"140f07c56050c437725723d0b355dcec8a43797ca37342c8aa67d23d253e901f"}
Sep 09 21:00:24.267556 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:24.267517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerStarted","Data":"256bfc5a4fd4c1badd6433b86da85c52ec9a90ba8a4aace3f8ec7061f66d8d73"}
Sep 09 21:00:24.268071 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:24.267578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerStarted","Data":"de8b626ae8a69f014cc0a45e282dc202a6686b90479baa1d56e45cffbe533055"}
Sep 09 21:00:24.268071 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:24.267784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:24.286892 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:24.286834    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podStartSLOduration=3.219418881 podStartE2EDuration="7.286819792s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.067400911s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:24.284598515 +0000 UTC m=+2951.323768488" watchObservedRunningTime="2026-09-09 21:00:24.286819792 +0000 UTC m=+2951.325989744"
Sep 09 21:00:25.272531 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:25.272497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:25.273794 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:25.273770    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:00:26.276551 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:26.276510    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:00:31.282564 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:31.282536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:00:31.283022 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:31.282995    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:00:41.283498 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:41.283458    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:00:46.602352 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.602311    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"]
Sep 09 21:00:46.603761 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.603731    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="storage-initializer"
Sep 09 21:00:46.603935 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.603923    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="storage-initializer"
Sep 09 21:00:46.604037 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.604027    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container"
Sep 09 21:00:46.604108 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.604099    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container"
Sep 09 21:00:46.604213 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.604204    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kube-rbac-proxy"
Sep 09 21:00:46.604288 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.604280    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kube-rbac-proxy"
Sep 09 21:00:46.604535 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.604523    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kserve-container"
Sep 09 21:00:46.604621 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.604612    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b8a4c8-f6fe-42d0-a715-64f35ea72d72" containerName="kube-rbac-proxy"
Sep 09 21:00:46.609596 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.609570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.612571 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.612549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 21:00:46.612775 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.612755    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 09 21:00:46.612878 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.612787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 09 21:00:46.619193 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.619163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"]
Sep 09 21:00:46.709817 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.709777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32511c08-eb78-4640-bdd0-3e29264149cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.709817 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.709817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32511c08-eb78-4640-bdd0-3e29264149cd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.710039 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.709861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32511c08-eb78-4640-bdd0-3e29264149cd-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.710039 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.709928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4b9\" (UniqueName: \"kubernetes.io/projected/32511c08-eb78-4640-bdd0-3e29264149cd-kube-api-access-vq4b9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.810884 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.810841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32511c08-eb78-4640-bdd0-3e29264149cd-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.811054 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.810959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4b9\" (UniqueName: \"kubernetes.io/projected/32511c08-eb78-4640-bdd0-3e29264149cd-kube-api-access-vq4b9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.811054 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.810991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32511c08-eb78-4640-bdd0-3e29264149cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.811054 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.811015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32511c08-eb78-4640-bdd0-3e29264149cd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.811431 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.811408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32511c08-eb78-4640-bdd0-3e29264149cd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.811750 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.811731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32511c08-eb78-4640-bdd0-3e29264149cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.813475 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.813457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32511c08-eb78-4640-bdd0-3e29264149cd-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.820520 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.820496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4b9\" (UniqueName: \"kubernetes.io/projected/32511c08-eb78-4640-bdd0-3e29264149cd-kube-api-access-vq4b9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:46.921460 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:46.921364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:47.200141 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:47.200114    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"]
Sep 09 21:00:47.201997 ip-10-0-129-130 kubenswrapper[2563]: W0909 21:00:47.201964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32511c08_eb78_4640_bdd0_3e29264149cd.slice/crio-2c106014f764eece1631746a681eeaee5a67e44badeaaa3379e7f71bb444fb6e WatchSource:0}: Error finding container 2c106014f764eece1631746a681eeaee5a67e44badeaaa3379e7f71bb444fb6e: Status 404 returned error can't find the container with id 2c106014f764eece1631746a681eeaee5a67e44badeaaa3379e7f71bb444fb6e
Sep 09 21:00:47.374539 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:47.374492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx" event={"ID":"32511c08-eb78-4640-bdd0-3e29264149cd","Type":"ContainerStarted","Data":"8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba"}
Sep 09 21:00:47.374742 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:47.374563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx" event={"ID":"32511c08-eb78-4640-bdd0-3e29264149cd","Type":"ContainerStarted","Data":"2c106014f764eece1631746a681eeaee5a67e44badeaaa3379e7f71bb444fb6e"}
Sep 09 21:00:49.386470 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:49.386445    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_32511c08-eb78-4640-bdd0-3e29264149cd/storage-initializer/0.log"
Sep 09 21:00:49.386901 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:49.386484    2563 generic.go:356] "Generic (PLEG): container finished" podID="32511c08-eb78-4640-bdd0-3e29264149cd" containerID="8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba" exitCode=1
Sep 09 21:00:49.386901 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:49.386518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx" event={"ID":"32511c08-eb78-4640-bdd0-3e29264149cd","Type":"ContainerDied","Data":"8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba"}
Sep 09 21:00:50.393357 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:50.393328    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_32511c08-eb78-4640-bdd0-3e29264149cd/storage-initializer/0.log"
Sep 09 21:00:50.393780 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:50.393426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx" event={"ID":"32511c08-eb78-4640-bdd0-3e29264149cd","Type":"ContainerStarted","Data":"03640a3a681ca46e620603b8a3093947ec04fae4d700d1ba642624a1ccf4f7cc"}
Sep 09 21:00:51.282996 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:51.282961    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:00:56.423021 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:56.422991    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_32511c08-eb78-4640-bdd0-3e29264149cd/storage-initializer/1.log"
Sep 09 21:00:56.423458 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:56.423316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_32511c08-eb78-4640-bdd0-3e29264149cd/storage-initializer/0.log"
Sep 09 21:00:56.423458 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:56.423343    2563 generic.go:356] "Generic (PLEG): container finished" podID="32511c08-eb78-4640-bdd0-3e29264149cd" containerID="03640a3a681ca46e620603b8a3093947ec04fae4d700d1ba642624a1ccf4f7cc" exitCode=1
Sep 09 21:00:56.423458 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:56.423422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx" event={"ID":"32511c08-eb78-4640-bdd0-3e29264149cd","Type":"ContainerDied","Data":"03640a3a681ca46e620603b8a3093947ec04fae4d700d1ba642624a1ccf4f7cc"}
Sep 09 21:00:56.423562 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:56.423484    2563 scope.go:117] "RemoveContainer" containerID="8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba"
Sep 09 21:00:56.423807 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:56.423792    2563 scope.go:117] "RemoveContainer" containerID="8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba"
Sep 09 21:00:56.436757 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:56.436717    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_kserve-ci-e2e-test_32511c08-eb78-4640-bdd0-3e29264149cd_0 in pod sandbox 2c106014f764eece1631746a681eeaee5a67e44badeaaa3379e7f71bb444fb6e from index: no such id: '8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba'" containerID="8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba"
Sep 09 21:00:56.436881 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:56.436783    2563 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_kserve-ci-e2e-test_32511c08-eb78-4640-bdd0-3e29264149cd_0 in pod sandbox 2c106014f764eece1631746a681eeaee5a67e44badeaaa3379e7f71bb444fb6e from index: no such id: '8d4a8b25862d7115faa71b72f162859ce9e00d6055dd1e531e52efd205ff22ba'; Skipping pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_kserve-ci-e2e-test(32511c08-eb78-4640-bdd0-3e29264149cd)\"" logger="UnhandledError"
Sep 09 21:00:56.437060 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:00:56.437029    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_kserve-ci-e2e-test(32511c08-eb78-4640-bdd0-3e29264149cd)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx" podUID="32511c08-eb78-4640-bdd0-3e29264149cd"
Sep 09 21:00:56.577506 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:56.577468    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"]
Sep 09 21:00:57.431237 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.431211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_32511c08-eb78-4640-bdd0-3e29264149cd/storage-initializer/1.log"
Sep 09 21:00:57.559284 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.559262    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_32511c08-eb78-4640-bdd0-3e29264149cd/storage-initializer/1.log"
Sep 09 21:00:57.562641 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.562625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:57.705069 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.704998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq4b9\" (UniqueName: \"kubernetes.io/projected/32511c08-eb78-4640-bdd0-3e29264149cd-kube-api-access-vq4b9\") pod \"32511c08-eb78-4640-bdd0-3e29264149cd\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") "
Sep 09 21:00:57.705284 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.705080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32511c08-eb78-4640-bdd0-3e29264149cd-kube-rbac-proxy-sar-config\") pod \"32511c08-eb78-4640-bdd0-3e29264149cd\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") "
Sep 09 21:00:57.705284 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.705124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32511c08-eb78-4640-bdd0-3e29264149cd-kserve-provision-location\") pod \"32511c08-eb78-4640-bdd0-3e29264149cd\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") "
Sep 09 21:00:57.705284 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.705156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32511c08-eb78-4640-bdd0-3e29264149cd-proxy-tls\") pod \"32511c08-eb78-4640-bdd0-3e29264149cd\" (UID: \"32511c08-eb78-4640-bdd0-3e29264149cd\") "
Sep 09 21:00:57.705466 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.705403    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32511c08-eb78-4640-bdd0-3e29264149cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32511c08-eb78-4640-bdd0-3e29264149cd" (UID: "32511c08-eb78-4640-bdd0-3e29264149cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:57.705517 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.705483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32511c08-eb78-4640-bdd0-3e29264149cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32511c08-eb78-4640-bdd0-3e29264149cd" (UID: "32511c08-eb78-4640-bdd0-3e29264149cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:57.707505 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.707474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32511c08-eb78-4640-bdd0-3e29264149cd-kube-api-access-vq4b9" (OuterVolumeSpecName: "kube-api-access-vq4b9") pod "32511c08-eb78-4640-bdd0-3e29264149cd" (UID: "32511c08-eb78-4640-bdd0-3e29264149cd"). InnerVolumeSpecName "kube-api-access-vq4b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:57.707611 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.707573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32511c08-eb78-4640-bdd0-3e29264149cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32511c08-eb78-4640-bdd0-3e29264149cd" (UID: "32511c08-eb78-4640-bdd0-3e29264149cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:57.805862 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.805828    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4b9\" (UniqueName: \"kubernetes.io/projected/32511c08-eb78-4640-bdd0-3e29264149cd-kube-api-access-vq4b9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:57.805862 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.805854    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32511c08-eb78-4640-bdd0-3e29264149cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:57.805862 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.805864    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32511c08-eb78-4640-bdd0-3e29264149cd-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:57.805862 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:57.805874    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32511c08-eb78-4640-bdd0-3e29264149cd-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:00:58.436631 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:58.436601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx_32511c08-eb78-4640-bdd0-3e29264149cd/storage-initializer/1.log"
Sep 09 21:00:58.437056 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:58.436698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx" event={"ID":"32511c08-eb78-4640-bdd0-3e29264149cd","Type":"ContainerDied","Data":"2c106014f764eece1631746a681eeaee5a67e44badeaaa3379e7f71bb444fb6e"}
Sep 09 21:00:58.437056 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:58.436728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"
Sep 09 21:00:58.437056 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:58.436768    2563 scope.go:117] "RemoveContainer" containerID="03640a3a681ca46e620603b8a3093947ec04fae4d700d1ba642624a1ccf4f7cc"
Sep 09 21:00:58.473369 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:58.473338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"]
Sep 09 21:00:58.477573 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:58.477549    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6857f58768-6j7tx"]
Sep 09 21:00:59.424322 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:00:59.424288    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32511c08-eb78-4640-bdd0-3e29264149cd" path="/var/lib/kubelet/pods/32511c08-eb78-4640-bdd0-3e29264149cd/volumes"
Sep 09 21:01:01.283566 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:01.283527    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:01:11.283858 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:11.283819    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:01:21.283667 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:21.283604    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:01:31.283782 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:31.283738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:01:41.284413 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:41.284371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:01:48.056842 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:48.056807    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"]
Sep 09 21:01:48.057311 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:48.057063    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" containerID="cri-o://de8b626ae8a69f014cc0a45e282dc202a6686b90479baa1d56e45cffbe533055" gracePeriod=30
Sep 09 21:01:48.057311 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:48.057091    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kube-rbac-proxy" containerID="cri-o://256bfc5a4fd4c1badd6433b86da85c52ec9a90ba8a4aace3f8ec7061f66d8d73" gracePeriod=30
Sep 09 21:01:48.666737 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:48.666702    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerID="256bfc5a4fd4c1badd6433b86da85c52ec9a90ba8a4aace3f8ec7061f66d8d73" exitCode=2
Sep 09 21:01:48.666925 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:48.666746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerDied","Data":"256bfc5a4fd4c1badd6433b86da85c52ec9a90ba8a4aace3f8ec7061f66d8d73"}
Sep 09 21:01:51.277412 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:51.277366    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 09 21:01:51.283144 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:51.283118    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 21:01:53.692165 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.692133    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerID="de8b626ae8a69f014cc0a45e282dc202a6686b90479baa1d56e45cffbe533055" exitCode=0
Sep 09 21:01:53.692529 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.692205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerDied","Data":"de8b626ae8a69f014cc0a45e282dc202a6686b90479baa1d56e45cffbe533055"}
Sep 09 21:01:53.811064 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.811037    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:01:53.881608 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.881579    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-rbac-proxy-sar-config\") pod \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") "
Sep 09 21:01:53.881802 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.881689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls\") pod \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") "
Sep 09 21:01:53.881802 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.881718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwcpx\" (UniqueName: \"kubernetes.io/projected/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-api-access-wwcpx\") pod \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") "
Sep 09 21:01:53.881802 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.881763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-cabundle-cert\") pod \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") "
Sep 09 21:01:53.881955 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.881897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kserve-provision-location\") pod \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\" (UID: \"7c218eb5-221d-49fc-bbf5-aa3d90ccebff\") "
Sep 09 21:01:53.882023 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.882001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c218eb5-221d-49fc-bbf5-aa3d90ccebff" (UID: "7c218eb5-221d-49fc-bbf5-aa3d90ccebff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:01:53.882090 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.882072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c218eb5-221d-49fc-bbf5-aa3d90ccebff" (UID: "7c218eb5-221d-49fc-bbf5-aa3d90ccebff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:01:53.882259 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.882230    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c218eb5-221d-49fc-bbf5-aa3d90ccebff" (UID: "7c218eb5-221d-49fc-bbf5-aa3d90ccebff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:53.882555 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.882294    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:01:53.882555 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.882308    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:01:53.883937 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.883919    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c218eb5-221d-49fc-bbf5-aa3d90ccebff" (UID: "7c218eb5-221d-49fc-bbf5-aa3d90ccebff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:01:53.884028 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.883998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-api-access-wwcpx" (OuterVolumeSpecName: "kube-api-access-wwcpx") pod "7c218eb5-221d-49fc-bbf5-aa3d90ccebff" (UID: "7c218eb5-221d-49fc-bbf5-aa3d90ccebff"). InnerVolumeSpecName "kube-api-access-wwcpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:53.982968 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.982867    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:01:53.982968 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.982909    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwcpx\" (UniqueName: \"kubernetes.io/projected/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kube-api-access-wwcpx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:01:53.982968 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:53.982920    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c218eb5-221d-49fc-bbf5-aa3d90ccebff-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:01:54.698284 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:54.698251    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"
Sep 09 21:01:54.698720 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:54.698251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf" event={"ID":"7c218eb5-221d-49fc-bbf5-aa3d90ccebff","Type":"ContainerDied","Data":"942e37c7371ee0c3489f77e94ebe3d8aa04e2a59a1220e6446600e4e3b1a9940"}
Sep 09 21:01:54.698720 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:54.698406    2563 scope.go:117] "RemoveContainer" containerID="256bfc5a4fd4c1badd6433b86da85c52ec9a90ba8a4aace3f8ec7061f66d8d73"
Sep 09 21:01:54.707659 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:54.707625    2563 scope.go:117] "RemoveContainer" containerID="de8b626ae8a69f014cc0a45e282dc202a6686b90479baa1d56e45cffbe533055"
Sep 09 21:01:54.715582 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:54.715554    2563 scope.go:117] "RemoveContainer" containerID="140f07c56050c437725723d0b355dcec8a43797ca37342c8aa67d23d253e901f"
Sep 09 21:01:54.720461 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:54.720438    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"]
Sep 09 21:01:54.724327 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:54.724221    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-bdqzf"]
Sep 09 21:01:55.425454 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:01:55.425417    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" path="/var/lib/kubelet/pods/7c218eb5-221d-49fc-bbf5-aa3d90ccebff/volumes"
Sep 09 21:02:13.983379 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983337    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"]
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983685    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kube-rbac-proxy"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983698    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kube-rbac-proxy"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983715    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983721    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983731    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32511c08-eb78-4640-bdd0-3e29264149cd" containerName="storage-initializer"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983738    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32511c08-eb78-4640-bdd0-3e29264149cd" containerName="storage-initializer"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983749    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="storage-initializer"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983754    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="storage-initializer"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983801    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kserve-container"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983809    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c218eb5-221d-49fc-bbf5-aa3d90ccebff" containerName="kube-rbac-proxy"
Sep 09 21:02:13.983823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983816    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32511c08-eb78-4640-bdd0-3e29264149cd" containerName="storage-initializer"
Sep 09 21:02:13.984168 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32511c08-eb78-4640-bdd0-3e29264149cd" containerName="storage-initializer"
Sep 09 21:02:13.984168 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983883    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32511c08-eb78-4640-bdd0-3e29264149cd" containerName="storage-initializer"
Sep 09 21:02:13.984168 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.983926    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32511c08-eb78-4640-bdd0-3e29264149cd" containerName="storage-initializer"
Sep 09 21:02:13.988696 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.988637    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:13.991534 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.991482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:02:13.991534 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.991489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:02:13.991534 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.991524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 09 21:02:13.992620 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.992603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:02:13.992759 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.992737    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:02:13.992876 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.992851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 21:02:13.992952 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.992851    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 21:02:13.997602 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:13.997579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"]
Sep 09 21:02:14.160572 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.160539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5de0c6a-c00d-4062-8fba-8956448ea03d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.160572 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.160578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.160826 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.160603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5de0c6a-c00d-4062-8fba-8956448ea03d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.160826 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.160720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9784\" (UniqueName: \"kubernetes.io/projected/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-api-access-m9784\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.160826 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.160764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.261552 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.261500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5de0c6a-c00d-4062-8fba-8956448ea03d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.261552 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.261557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.261847 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.261579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5de0c6a-c00d-4062-8fba-8956448ea03d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.261847 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.261622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9784\" (UniqueName: \"kubernetes.io/projected/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-api-access-m9784\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.261847 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.261679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.262014 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.261990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5de0c6a-c00d-4062-8fba-8956448ea03d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.262257 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.262235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.262378 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.262360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.264345 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.264320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5de0c6a-c00d-4062-8fba-8956448ea03d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.271015 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.270992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9784\" (UniqueName: \"kubernetes.io/projected/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-api-access-m9784\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.301433 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.301402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:14.575473 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.575450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"]
Sep 09 21:02:14.577366 ip-10-0-129-130 kubenswrapper[2563]: W0909 21:02:14.577344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5de0c6a_c00d_4062_8fba_8956448ea03d.slice/crio-2e649387812329dbeffda3491a019320c0153c8df1ce2bd838a374976d9fe4c9 WatchSource:0}: Error finding container 2e649387812329dbeffda3491a019320c0153c8df1ce2bd838a374976d9fe4c9: Status 404 returned error can't find the container with id 2e649387812329dbeffda3491a019320c0153c8df1ce2bd838a374976d9fe4c9
Sep 09 21:02:14.788156 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.788120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerStarted","Data":"87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2"}
Sep 09 21:02:14.788372 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:14.788194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerStarted","Data":"2e649387812329dbeffda3491a019320c0153c8df1ce2bd838a374976d9fe4c9"}
Sep 09 21:02:19.812562 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:19.812523    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerID="87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2" exitCode=0
Sep 09 21:02:19.812972 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:19.812575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerDied","Data":"87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2"}
Sep 09 21:02:30.876201 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:30.876170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerStarted","Data":"843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39"}
Sep 09 21:02:31.882812 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:31.882766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerStarted","Data":"9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66"}
Sep 09 21:02:31.883342 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:31.882899    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:31.883342 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:31.882962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:31.884518 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:31.884491    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 09 21:02:31.903876 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:31.903815    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podStartSLOduration=3.438258482 podStartE2EDuration="18.903798571s" totalImagesPullingTime="10.964918754s" totalInitContainerRuntime="4.500621335s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:31.901543832 +0000 UTC m=+3078.940713784" watchObservedRunningTime="2026-09-09 21:02:31.903798571 +0000 UTC m=+3078.942968522"
Sep 09 21:02:32.886982 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:32.886945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 09 21:02:37.893228 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:37.893196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:02:37.893756 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:37.893730    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 09 21:02:47.894153 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:47.894110    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 09 21:02:57.894303 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:02:57.894263    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 09 21:03:07.894139 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:07.894094    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 09 21:03:17.894831 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:17.894800    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:03:25.521807 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:25.521772    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"]
Sep 09 21:03:25.522263 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:25.522006    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" containerID="cri-o://843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39" gracePeriod=30
Sep 09 21:03:25.522263 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:25.522088    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kube-rbac-proxy" containerID="cri-o://9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66" gracePeriod=30
Sep 09 21:03:26.134980 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:26.134943    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerID="9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66" exitCode=2
Sep 09 21:03:26.135157 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:26.135011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerDied","Data":"9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66"}
Sep 09 21:03:27.887668 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:27.887598    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 09 21:03:27.893871 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:27.893842    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 09 21:03:28.572781 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.572753    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:03:28.700167 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.700074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-rbac-proxy-sar-config\") pod \"e5de0c6a-c00d-4062-8fba-8956448ea03d\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") "
Sep 09 21:03:28.700167 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.700144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-cabundle-cert\") pod \"e5de0c6a-c00d-4062-8fba-8956448ea03d\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") "
Sep 09 21:03:28.700406 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.700181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5de0c6a-c00d-4062-8fba-8956448ea03d-kserve-provision-location\") pod \"e5de0c6a-c00d-4062-8fba-8956448ea03d\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") "
Sep 09 21:03:28.700406 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.700260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9784\" (UniqueName: \"kubernetes.io/projected/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-api-access-m9784\") pod \"e5de0c6a-c00d-4062-8fba-8956448ea03d\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") "
Sep 09 21:03:28.700406 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.700310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5de0c6a-c00d-4062-8fba-8956448ea03d-proxy-tls\") pod \"e5de0c6a-c00d-4062-8fba-8956448ea03d\" (UID: \"e5de0c6a-c00d-4062-8fba-8956448ea03d\") "
Sep 09 21:03:28.700541 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.700508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5de0c6a-c00d-4062-8fba-8956448ea03d" (UID: "e5de0c6a-c00d-4062-8fba-8956448ea03d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:03:28.700579 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.700517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5de0c6a-c00d-4062-8fba-8956448ea03d" (UID: "e5de0c6a-c00d-4062-8fba-8956448ea03d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:03:28.702528 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.702488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-api-access-m9784" (OuterVolumeSpecName: "kube-api-access-m9784") pod "e5de0c6a-c00d-4062-8fba-8956448ea03d" (UID: "e5de0c6a-c00d-4062-8fba-8956448ea03d"). InnerVolumeSpecName "kube-api-access-m9784". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:03:28.702682 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.702587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5de0c6a-c00d-4062-8fba-8956448ea03d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5de0c6a-c00d-4062-8fba-8956448ea03d" (UID: "e5de0c6a-c00d-4062-8fba-8956448ea03d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:28.708739 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.708713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5de0c6a-c00d-4062-8fba-8956448ea03d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5de0c6a-c00d-4062-8fba-8956448ea03d" (UID: "e5de0c6a-c00d-4062-8fba-8956448ea03d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:28.800935 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.800898    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9784\" (UniqueName: \"kubernetes.io/projected/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-api-access-m9784\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:03:28.800935 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.800927    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5de0c6a-c00d-4062-8fba-8956448ea03d-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:03:28.800935 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.800936    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:03:28.800935 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.800946    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5de0c6a-c00d-4062-8fba-8956448ea03d-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:03:28.801169 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:28.800955    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5de0c6a-c00d-4062-8fba-8956448ea03d-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:03:29.152499 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.152461    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerID="843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39" exitCode=0
Sep 09 21:03:29.152993 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.152515    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"
Sep 09 21:03:29.152993 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.152560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerDied","Data":"843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39"}
Sep 09 21:03:29.152993 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.152626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z" event={"ID":"e5de0c6a-c00d-4062-8fba-8956448ea03d","Type":"ContainerDied","Data":"2e649387812329dbeffda3491a019320c0153c8df1ce2bd838a374976d9fe4c9"}
Sep 09 21:03:29.152993 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.152678    2563 scope.go:117] "RemoveContainer" containerID="9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66"
Sep 09 21:03:29.161943 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.161922    2563 scope.go:117] "RemoveContainer" containerID="843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39"
Sep 09 21:03:29.169972 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.169953    2563 scope.go:117] "RemoveContainer" containerID="87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2"
Sep 09 21:03:29.177184 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.177156    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"]
Sep 09 21:03:29.178394 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.178377    2563 scope.go:117] "RemoveContainer" containerID="9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66"
Sep 09 21:03:29.178743 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:03:29.178722    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66\": container with ID starting with 9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66 not found: ID does not exist" containerID="9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66"
Sep 09 21:03:29.178814 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.178755    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66"} err="failed to get container status \"9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66\": rpc error: code = NotFound desc = could not find container \"9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66\": container with ID starting with 9d2f5f02a8eecd2992b22a7b3bdd8d867ca41a87c27b8a2127f5636f55d1da66 not found: ID does not exist"
Sep 09 21:03:29.178814 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.178778    2563 scope.go:117] "RemoveContainer" containerID="843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39"
Sep 09 21:03:29.179052 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:03:29.179037    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39\": container with ID starting with 843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39 not found: ID does not exist" containerID="843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39"
Sep 09 21:03:29.179095 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.179056    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39"} err="failed to get container status \"843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39\": rpc error: code = NotFound desc = could not find container \"843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39\": container with ID starting with 843904c096794720b9bbeffafb007fb6ff88fc992b5c3b953ca42fd37d659f39 not found: ID does not exist"
Sep 09 21:03:29.179095 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.179068    2563 scope.go:117] "RemoveContainer" containerID="87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2"
Sep 09 21:03:29.179315 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:03:29.179296    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2\": container with ID starting with 87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2 not found: ID does not exist" containerID="87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2"
Sep 09 21:03:29.179372 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.179323    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2"} err="failed to get container status \"87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2\": rpc error: code = NotFound desc = could not find container \"87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2\": container with ID starting with 87a93bce600890a8a8f68be7db11fe88a3c61026a1629e1c42b4c0d6e2637ae2 not found: ID does not exist"
Sep 09 21:03:29.184975 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.184949    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-hf45z"]
Sep 09 21:03:29.425464 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:03:29.425370    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" path="/var/lib/kubelet/pods/e5de0c6a-c00d-4062-8fba-8956448ea03d/volumes"
Sep 09 21:04:19.142143 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142107    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"]
Sep 09 21:04:19.142577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142464    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container"
Sep 09 21:04:19.142577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container"
Sep 09 21:04:19.142577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142492    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kube-rbac-proxy"
Sep 09 21:04:19.142577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142497    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kube-rbac-proxy"
Sep 09 21:04:19.142577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142506    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="storage-initializer"
Sep 09 21:04:19.142577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142512    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="storage-initializer"
Sep 09 21:04:19.142577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142569    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kserve-container"
Sep 09 21:04:19.142836 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.142582    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5de0c6a-c00d-4062-8fba-8956448ea03d" containerName="kube-rbac-proxy"
Sep 09 21:04:19.145400 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.145382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.148371 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.148344    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:04:19.148509 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.148344    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 09 21:04:19.148509 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.148380    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:04:19.148509 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.148340    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 21:04:19.148715 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.148693    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:04:19.148909 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.148893    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 09 21:04:19.149361 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.149344    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:04:19.163871 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.163840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"]
Sep 09 21:04:19.299181 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.299126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njhl7\" (UniqueName: \"kubernetes.io/projected/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-api-access-njhl7\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.299388 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.299189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.299388 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.299273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.299388 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.299309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.299388 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.299338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.400431 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.400333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.400431 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.400379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.400431 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.400408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.400776 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.400456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njhl7\" (UniqueName: \"kubernetes.io/projected/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-api-access-njhl7\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.400776 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.400482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.400895 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.400793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.401233 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.401208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.401233 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.401229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.403058 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.403026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.409695 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.409642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njhl7\" (UniqueName: \"kubernetes.io/projected/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-api-access-njhl7\") pod \"isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.455776 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.455733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:19.732226 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:19.732190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"]
Sep 09 21:04:19.734638 ip-10-0-129-130 kubenswrapper[2563]: W0909 21:04:19.734608    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f25e9ee_1d98_411f_99cd_8e0ac56e98bf.slice/crio-42c48828b439d3c644a5700eee6d77b01ae1100c633fb423fd7ec7a9272e0c09 WatchSource:0}: Error finding container 42c48828b439d3c644a5700eee6d77b01ae1100c633fb423fd7ec7a9272e0c09: Status 404 returned error can't find the container with id 42c48828b439d3c644a5700eee6d77b01ae1100c633fb423fd7ec7a9272e0c09
Sep 09 21:04:20.381926 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:20.381887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerStarted","Data":"beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621"}
Sep 09 21:04:20.382310 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:20.381958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerStarted","Data":"42c48828b439d3c644a5700eee6d77b01ae1100c633fb423fd7ec7a9272e0c09"}
Sep 09 21:04:24.401105 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:24.401072    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerID="beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621" exitCode=0
Sep 09 21:04:24.401492 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:24.401125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerDied","Data":"beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621"}
Sep 09 21:04:24.474342 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:24.474324    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:04:25.411178 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:25.411138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerStarted","Data":"2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede"}
Sep 09 21:04:28.430491 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:28.430449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerStarted","Data":"be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158"}
Sep 09 21:04:28.430949 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:28.430674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:28.430949 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:28.430721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:28.430949 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:28.430747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerStarted","Data":"2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa"}
Sep 09 21:04:28.460971 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:28.460907    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podStartSLOduration=2.530463824 podStartE2EDuration="9.460887011s" totalImagesPullingTime="2.996744039s" totalInitContainerRuntime="3.933679148s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:28.458851225 +0000 UTC m=+3195.498021185" watchObservedRunningTime="2026-09-09 21:04:28.460887011 +0000 UTC m=+3195.500056963"
Sep 09 21:04:29.434788 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:29.434749    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:35.445027 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:35.444994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:04:55.446373 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:55.446331    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 21:04:59.457021 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:04:59.456962    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 21:05:05.449150 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:05.449063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:05:35.450168 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:35.450136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:05:39.390582 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:39.390544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"]
Sep 09 21:05:39.391030 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:39.390849    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container" containerID="cri-o://2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede" gracePeriod=30
Sep 09 21:05:39.391030 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:39.390898    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-agent" containerID="cri-o://2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa" gracePeriod=30
Sep 09 21:05:39.391030 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:39.390901    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy" containerID="cri-o://be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158" gracePeriod=30
Sep 09 21:05:39.755688 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:39.755590    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerID="be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158" exitCode=2
Sep 09 21:05:39.755688 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:39.755674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerDied","Data":"be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158"}
Sep 09 21:05:40.440318 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:40.440255    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 09 21:05:41.767885 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:41.767852    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerID="2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede" exitCode=0
Sep 09 21:05:41.768244 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:41.767928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerDied","Data":"2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede"}
Sep 09 21:05:45.440390 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:45.440347    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 09 21:05:45.445823 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:45.445794    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 21:05:50.440215 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:50.440165    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 09 21:05:50.440622 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:50.440307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:05:55.439792 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:55.439749    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 09 21:05:55.446240 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:05:55.446210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 21:06:00.440424 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:00.440377    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 09 21:06:05.440328 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:05.440283    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 09 21:06:05.445801 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:05.445772    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 21:06:05.445907 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:05.445894    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:06:09.575059 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.575034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:06:09.590581 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.590554    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njhl7\" (UniqueName: \"kubernetes.io/projected/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-api-access-njhl7\") pod \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") "
Sep 09 21:06:09.590741 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.590595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kserve-provision-location\") pod \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") "
Sep 09 21:06:09.590741 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.590624    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-proxy-tls\") pod \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") "
Sep 09 21:06:09.590872 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.590800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-cabundle-cert\") pod \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") "
Sep 09 21:06:09.590937 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.590870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-rbac-proxy-sar-config\") pod \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\" (UID: \"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf\") "
Sep 09 21:06:09.590937 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.590879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" (UID: "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:09.591180 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.591159    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:06:09.591419 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.591171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" (UID: "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:09.591419 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.591224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" (UID: "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:09.592828 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.592810    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" (UID: "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:09.592921 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.592897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-api-access-njhl7" (OuterVolumeSpecName: "kube-api-access-njhl7") pod "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" (UID: "6f25e9ee-1d98-411f-99cd-8e0ac56e98bf"). InnerVolumeSpecName "kube-api-access-njhl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:09.692523 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.692440    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:06:09.692523 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.692472    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:06:09.692523 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.692482    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njhl7\" (UniqueName: \"kubernetes.io/projected/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-kube-api-access-njhl7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:06:09.692523 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.692491    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:06:09.894889 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.894854    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerID="2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa" exitCode=137
Sep 09 21:06:09.895053 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.894922    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"
Sep 09 21:06:09.895053 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.894927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerDied","Data":"2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa"}
Sep 09 21:06:09.895053 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.895051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz" event={"ID":"6f25e9ee-1d98-411f-99cd-8e0ac56e98bf","Type":"ContainerDied","Data":"42c48828b439d3c644a5700eee6d77b01ae1100c633fb423fd7ec7a9272e0c09"}
Sep 09 21:06:09.895209 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.895082    2563 scope.go:117] "RemoveContainer" containerID="be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158"
Sep 09 21:06:09.904139 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.904122    2563 scope.go:117] "RemoveContainer" containerID="2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa"
Sep 09 21:06:09.911601 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.911583    2563 scope.go:117] "RemoveContainer" containerID="2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede"
Sep 09 21:06:09.918405 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.918382    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"]
Sep 09 21:06:09.919459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.919444    2563 scope.go:117] "RemoveContainer" containerID="beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621"
Sep 09 21:06:09.922373 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.922355    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c7ccf85b8-n4xhz"]
Sep 09 21:06:09.927444 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.927424    2563 scope.go:117] "RemoveContainer" containerID="be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158"
Sep 09 21:06:09.927748 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:06:09.927729    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158\": container with ID starting with be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158 not found: ID does not exist" containerID="be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158"
Sep 09 21:06:09.927812 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.927758    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158"} err="failed to get container status \"be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158\": rpc error: code = NotFound desc = could not find container \"be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158\": container with ID starting with be5f92ce217e3a7eefb550192d557f2e4a6baf798eb95c72aa100f6589e6f158 not found: ID does not exist"
Sep 09 21:06:09.927812 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.927776    2563 scope.go:117] "RemoveContainer" containerID="2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa"
Sep 09 21:06:09.928058 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:06:09.928040    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa\": container with ID starting with 2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa not found: ID does not exist" containerID="2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa"
Sep 09 21:06:09.928111 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.928064    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa"} err="failed to get container status \"2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa\": rpc error: code = NotFound desc = could not find container \"2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa\": container with ID starting with 2b2569058f1ad0fabecc3e4236a5792d48127f6c21e8ec6c3e2e3ae4cf49ddfa not found: ID does not exist"
Sep 09 21:06:09.928111 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.928079    2563 scope.go:117] "RemoveContainer" containerID="2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede"
Sep 09 21:06:09.928313 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:06:09.928297    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede\": container with ID starting with 2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede not found: ID does not exist" containerID="2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede"
Sep 09 21:06:09.928354 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.928321    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede"} err="failed to get container status \"2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede\": rpc error: code = NotFound desc = could not find container \"2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede\": container with ID starting with 2215c58a88f4baac1fe248ae8d061d975b0a66af0ac06bf5affb7b6e47c3cede not found: ID does not exist"
Sep 09 21:06:09.928354 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.928336    2563 scope.go:117] "RemoveContainer" containerID="beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621"
Sep 09 21:06:09.928562 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:06:09.928547    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621\": container with ID starting with beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621 not found: ID does not exist" containerID="beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621"
Sep 09 21:06:09.928614 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:09.928564    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621"} err="failed to get container status \"beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621\": rpc error: code = NotFound desc = could not find container \"beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621\": container with ID starting with beb1c5d56b73d2721419a3e524c8c54d7fd9a909af8a5c3eda025404bf362621 not found: ID does not exist"
Sep 09 21:06:11.423974 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:11.423929    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" path="/var/lib/kubelet/pods/6f25e9ee-1d98-411f-99cd-8e0ac56e98bf/volumes"
Sep 09 21:06:50.901977 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.901946    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"]
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902305    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="storage-initializer"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902316    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="storage-initializer"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902325    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902331    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902350    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902356    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902364    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-agent"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-agent"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902415    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-agent"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902425    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kserve-container"
Sep 09 21:06:50.902459 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.902433    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f25e9ee-1d98-411f-99cd-8e0ac56e98bf" containerName="kube-rbac-proxy"
Sep 09 21:06:50.905614 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.905594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:50.908602 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.908579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:06:50.908790 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.908770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:06:50.908892 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.908863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f4eae3-kube-rbac-proxy-sar-config\""
Sep 09 21:06:50.908970 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.908901    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:06:50.910224 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.910191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f4eae3-predictor-serving-cert\""
Sep 09 21:06:50.910224 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.910218    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 21:06:50.910372 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.910247    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:06:50.917074 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:50.917050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"]
Sep 09 21:06:51.048899 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.048852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-cabundle-cert\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.048899 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.048894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjqb\" (UniqueName: \"kubernetes.io/projected/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-api-access-xmjqb\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.049249 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.048948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.049249 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.049002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kserve-provision-location\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.049249 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.049058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ba89df-ecab-41a9-abbc-9b4002d8d488-proxy-tls\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.149868 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.149835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-cabundle-cert\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.150056 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.149878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjqb\" (UniqueName: \"kubernetes.io/projected/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-api-access-xmjqb\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.150056 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.149918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.150056 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.149954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kserve-provision-location\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.150056 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.149976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ba89df-ecab-41a9-abbc-9b4002d8d488-proxy-tls\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.150386 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.150360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kserve-provision-location\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.150594 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.150573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-cabundle-cert\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.150689 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.150594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.152534 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.152494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ba89df-ecab-41a9-abbc-9b4002d8d488-proxy-tls\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.158477 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.158453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjqb\" (UniqueName: \"kubernetes.io/projected/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-api-access-xmjqb\") pod \"isvc-primary-f4eae3-predictor-579977fd97-xt84g\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") " pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.217345 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.217316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:51.491793 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:51.491769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"]
Sep 09 21:06:51.493851 ip-10-0-129-130 kubenswrapper[2563]: W0909 21:06:51.493826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ba89df_ecab_41a9_abbc_9b4002d8d488.slice/crio-419415e8bf425891ad844d10b34e29fa0c3921d465064d491fd480a08034c4cf WatchSource:0}: Error finding container 419415e8bf425891ad844d10b34e29fa0c3921d465064d491fd480a08034c4cf: Status 404 returned error can't find the container with id 419415e8bf425891ad844d10b34e29fa0c3921d465064d491fd480a08034c4cf
Sep 09 21:06:52.080345 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:52.080302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerStarted","Data":"f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503"}
Sep 09 21:06:52.080852 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:52.080361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerStarted","Data":"419415e8bf425891ad844d10b34e29fa0c3921d465064d491fd480a08034c4cf"}
Sep 09 21:06:56.100410 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:56.100369    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerID="f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503" exitCode=0
Sep 09 21:06:56.100951 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:56.100423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerDied","Data":"f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503"}
Sep 09 21:06:57.107880 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:57.107841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerStarted","Data":"498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06"}
Sep 09 21:06:57.108272 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:57.107906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerStarted","Data":"cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0"}
Sep 09 21:06:57.108272 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:57.108085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:57.129554 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:57.129503    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podStartSLOduration=3.232423188 podStartE2EDuration="7.129489267s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.897066079s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:57.127802069 +0000 UTC m=+3344.166972021" watchObservedRunningTime="2026-09-09 21:06:57.129489267 +0000 UTC m=+3344.168659217"
Sep 09 21:06:58.113016 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:58.112984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:06:58.114249 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:58.114221    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:06:59.117526 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:06:59.117482    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:07:04.122840 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:07:04.122800    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:07:04.123452 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:07:04.123421    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:07:14.123773 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:07:14.123724    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:07:24.123870 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:07:24.123826    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:07:34.123518 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:07:34.123479    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:07:44.124271 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:07:44.124229    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:07:54.123747 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:07:54.123705    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:08:04.124238 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:04.124199    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 21:08:14.124534 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:14.124498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:08:54.210552 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:54.210520    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"]
Sep 09 21:08:54.211135 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:54.210780    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container" containerID="cri-o://cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0" gracePeriod=30
Sep 09 21:08:54.211135 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:54.210820    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kube-rbac-proxy" containerID="cri-o://498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06" gracePeriod=30
Sep 09 21:08:54.623314 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:54.623280    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerID="498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06" exitCode=2
Sep 09 21:08:54.623480 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:54.623355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerDied","Data":"498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06"}
Sep 09 21:08:58.951275 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:58.951252    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:08:59.027955 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.027881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjqb\" (UniqueName: \"kubernetes.io/projected/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-api-access-xmjqb\") pod \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") "
Sep 09 21:08:59.027955 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.027918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ba89df-ecab-41a9-abbc-9b4002d8d488-proxy-tls\") pod \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") "
Sep 09 21:08:59.028146 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.027965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-rbac-proxy-sar-config\") pod \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") "
Sep 09 21:08:59.028146 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.027988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kserve-provision-location\") pod \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") "
Sep 09 21:08:59.028146 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.028021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-cabundle-cert\") pod \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\" (UID: \"c8ba89df-ecab-41a9-abbc-9b4002d8d488\") "
Sep 09 21:08:59.028356 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.028326    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8ba89df-ecab-41a9-abbc-9b4002d8d488" (UID: "c8ba89df-ecab-41a9-abbc-9b4002d8d488"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:59.028356 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.028349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8ba89df-ecab-41a9-abbc-9b4002d8d488" (UID: "c8ba89df-ecab-41a9-abbc-9b4002d8d488"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:59.028515 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.028390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8ba89df-ecab-41a9-abbc-9b4002d8d488" (UID: "c8ba89df-ecab-41a9-abbc-9b4002d8d488"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:59.030174 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.030156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-api-access-xmjqb" (OuterVolumeSpecName: "kube-api-access-xmjqb") pod "c8ba89df-ecab-41a9-abbc-9b4002d8d488" (UID: "c8ba89df-ecab-41a9-abbc-9b4002d8d488"). InnerVolumeSpecName "kube-api-access-xmjqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:59.030246 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.030183    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ba89df-ecab-41a9-abbc-9b4002d8d488-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8ba89df-ecab-41a9-abbc-9b4002d8d488" (UID: "c8ba89df-ecab-41a9-abbc-9b4002d8d488"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:59.128958 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.128928    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjqb\" (UniqueName: \"kubernetes.io/projected/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-api-access-xmjqb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:08:59.128958 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.128951    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ba89df-ecab-41a9-abbc-9b4002d8d488-proxy-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:08:59.128958 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.128961    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:08:59.129174 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.128970    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ba89df-ecab-41a9-abbc-9b4002d8d488-kserve-provision-location\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:08:59.129174 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.128978    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ba89df-ecab-41a9-abbc-9b4002d8d488-cabundle-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\""
Sep 09 21:08:59.646305 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.646271    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerID="cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0" exitCode=0
Sep 09 21:08:59.646305 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.646308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerDied","Data":"cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0"}
Sep 09 21:08:59.646642 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.646343    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"
Sep 09 21:08:59.646642 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.646356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g" event={"ID":"c8ba89df-ecab-41a9-abbc-9b4002d8d488","Type":"ContainerDied","Data":"419415e8bf425891ad844d10b34e29fa0c3921d465064d491fd480a08034c4cf"}
Sep 09 21:08:59.646642 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.646384    2563 scope.go:117] "RemoveContainer" containerID="498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06"
Sep 09 21:08:59.656863 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.656844    2563 scope.go:117] "RemoveContainer" containerID="cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0"
Sep 09 21:08:59.665050 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.665021    2563 scope.go:117] "RemoveContainer" containerID="f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503"
Sep 09 21:08:59.665675 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.665638    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"]
Sep 09 21:08:59.672251 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.672228    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f4eae3-predictor-579977fd97-xt84g"]
Sep 09 21:08:59.673103 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.673082    2563 scope.go:117] "RemoveContainer" containerID="498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06"
Sep 09 21:08:59.673395 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:08:59.673372    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06\": container with ID starting with 498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06 not found: ID does not exist" containerID="498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06"
Sep 09 21:08:59.673461 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.673410    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06"} err="failed to get container status \"498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06\": rpc error: code = NotFound desc = could not find container \"498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06\": container with ID starting with 498f06c83e8ac463b9a2fee5b39af19c944ae1085444e6196f9b9d03e61e8a06 not found: ID does not exist"
Sep 09 21:08:59.673461 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.673436    2563 scope.go:117] "RemoveContainer" containerID="cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0"
Sep 09 21:08:59.673712 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:08:59.673692    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0\": container with ID starting with cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0 not found: ID does not exist" containerID="cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0"
Sep 09 21:08:59.673766 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.673720    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0"} err="failed to get container status \"cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0\": rpc error: code = NotFound desc = could not find container \"cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0\": container with ID starting with cee86e39a7fe7f75048af22245cdc06ee1fe4aca3504855b0b237f9c627a20b0 not found: ID does not exist"
Sep 09 21:08:59.673766 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.673736    2563 scope.go:117] "RemoveContainer" containerID="f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503"
Sep 09 21:08:59.673971 ip-10-0-129-130 kubenswrapper[2563]: E0909 21:08:59.673957    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503\": container with ID starting with f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503 not found: ID does not exist" containerID="f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503"
Sep 09 21:08:59.674010 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:08:59.673975    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503"} err="failed to get container status \"f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503\": rpc error: code = NotFound desc = could not find container \"f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503\": container with ID starting with f06ea89d0972e48d7ae708f17403e665c76c3f79ab8356c9075ad72a5f840503 not found: ID does not exist"
Sep 09 21:09:01.425677 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:01.425615    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" path="/var/lib/kubelet/pods/c8ba89df-ecab-41a9-abbc-9b4002d8d488/volumes"
Sep 09 21:09:39.939563 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.939524    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w9tpv/must-gather-w7xkr"]
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.939902    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kube-rbac-proxy"
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.939915    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kube-rbac-proxy"
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.939928    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="storage-initializer"
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.939936    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="storage-initializer"
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.939959    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container"
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.939966    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container"
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.940016    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kube-rbac-proxy"
Sep 09 21:09:39.940041 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.940029    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ba89df-ecab-41a9-abbc-9b4002d8d488" containerName="kserve-container"
Sep 09 21:09:39.943181 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.943165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:39.947333 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.947310    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w9tpv\"/\"default-dockercfg-7bnrw\""
Sep 09 21:09:39.948471 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.948450    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9tpv\"/\"openshift-service-ca.crt\""
Sep 09 21:09:39.948667 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.948618    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9tpv\"/\"kube-root-ca.crt\""
Sep 09 21:09:39.964073 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:39.964052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9tpv/must-gather-w7xkr"]
Sep 09 21:09:40.076069 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.076038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7b2\" (UniqueName: \"kubernetes.io/projected/e9106737-dee8-43e7-a1ea-46b39ea75875-kube-api-access-hp7b2\") pod \"must-gather-w7xkr\" (UID: \"e9106737-dee8-43e7-a1ea-46b39ea75875\") " pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:40.076260 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.076089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9106737-dee8-43e7-a1ea-46b39ea75875-must-gather-output\") pod \"must-gather-w7xkr\" (UID: \"e9106737-dee8-43e7-a1ea-46b39ea75875\") " pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:40.177476 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.177445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9106737-dee8-43e7-a1ea-46b39ea75875-must-gather-output\") pod \"must-gather-w7xkr\" (UID: \"e9106737-dee8-43e7-a1ea-46b39ea75875\") " pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:40.177675 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.177516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7b2\" (UniqueName: \"kubernetes.io/projected/e9106737-dee8-43e7-a1ea-46b39ea75875-kube-api-access-hp7b2\") pod \"must-gather-w7xkr\" (UID: \"e9106737-dee8-43e7-a1ea-46b39ea75875\") " pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:40.177824 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.177805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9106737-dee8-43e7-a1ea-46b39ea75875-must-gather-output\") pod \"must-gather-w7xkr\" (UID: \"e9106737-dee8-43e7-a1ea-46b39ea75875\") " pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:40.190205 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.190146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7b2\" (UniqueName: \"kubernetes.io/projected/e9106737-dee8-43e7-a1ea-46b39ea75875-kube-api-access-hp7b2\") pod \"must-gather-w7xkr\" (UID: \"e9106737-dee8-43e7-a1ea-46b39ea75875\") " pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:40.259834 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.259804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9tpv/must-gather-w7xkr"
Sep 09 21:09:40.538538 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.538459    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9tpv/must-gather-w7xkr"]
Sep 09 21:09:40.542372 ip-10-0-129-130 kubenswrapper[2563]: W0909 21:09:40.542344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9106737_dee8_43e7_a1ea_46b39ea75875.slice/crio-f6bf7039f73afe07755a3ae47298b069ea9a15b29472c92fe70b02d5b40d8593 WatchSource:0}: Error finding container f6bf7039f73afe07755a3ae47298b069ea9a15b29472c92fe70b02d5b40d8593: Status 404 returned error can't find the container with id f6bf7039f73afe07755a3ae47298b069ea9a15b29472c92fe70b02d5b40d8593
Sep 09 21:09:40.544477 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.544458    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:09:40.832933 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:40.832899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9tpv/must-gather-w7xkr" event={"ID":"e9106737-dee8-43e7-a1ea-46b39ea75875","Type":"ContainerStarted","Data":"f6bf7039f73afe07755a3ae47298b069ea9a15b29472c92fe70b02d5b40d8593"}
Sep 09 21:09:41.842252 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:41.841481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9tpv/must-gather-w7xkr" event={"ID":"e9106737-dee8-43e7-a1ea-46b39ea75875","Type":"ContainerStarted","Data":"1b52ee016508099a801cd4d9e69fa12dcbaa5bf407cf3621b28c6530a26b2527"}
Sep 09 21:09:41.842252 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:41.841544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9tpv/must-gather-w7xkr" event={"ID":"e9106737-dee8-43e7-a1ea-46b39ea75875","Type":"ContainerStarted","Data":"98b09f130efeb99caea38a51ee80118ec7ce843c31089244c76ae8e0d1e42f0e"}
Sep 09 21:09:41.860454 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:41.860404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w9tpv/must-gather-w7xkr" podStartSLOduration=2.082935415 podStartE2EDuration="2.860381104s" totalImagesPullingTime="777.445689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:41.859837383 +0000 UTC m=+3508.899007332" watchObservedRunningTime="2026-09-09 21:09:41.860381104 +0000 UTC m=+3508.899551055"
Sep 09 21:09:43.103808 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:43.103774    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gz6b2_e68f0176-5891-423d-ae93-8c6c8cdf1d93/global-pull-secret-syncer/0.log"
Sep 09 21:09:43.313078 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:43.313043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rcp2l_40e16b77-7730-4ae6-95a2-43bae3c36d86/konnectivity-agent/0.log"
Sep 09 21:09:43.339051 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:43.338998    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-130.ec2.internal_fcb022e37644c385755d64172316b214/haproxy/0.log"
Sep 09 21:09:47.723172 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:47.723135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ddf9a7ce-6192-4305-8822-b441a0427804/alertmanager/0.log"
Sep 09 21:09:47.756692 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:47.756601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ddf9a7ce-6192-4305-8822-b441a0427804/config-reloader/0.log"
Sep 09 21:09:47.788536 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:47.788505    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ddf9a7ce-6192-4305-8822-b441a0427804/kube-rbac-proxy-web/0.log"
Sep 09 21:09:47.817456 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:47.817428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ddf9a7ce-6192-4305-8822-b441a0427804/kube-rbac-proxy/0.log"
Sep 09 21:09:47.844751 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:47.844721    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ddf9a7ce-6192-4305-8822-b441a0427804/kube-rbac-proxy-metric/0.log"
Sep 09 21:09:47.878593 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:47.878519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ddf9a7ce-6192-4305-8822-b441a0427804/prom-label-proxy/0.log"
Sep 09 21:09:47.907880 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:47.907848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ddf9a7ce-6192-4305-8822-b441a0427804/init-config-reloader/0.log"
Sep 09 21:09:48.060936 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.060907    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56cf9559d6-hk8xp_cf95e321-da7d-4b72-8e98-5bd03a0b2176/metrics-server/0.log"
Sep 09 21:09:48.240755 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.240726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zgpc8_a22e518a-a8bf-4962-8c2f-1223a20c2114/node-exporter/0.log"
Sep 09 21:09:48.300889 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.300854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zgpc8_a22e518a-a8bf-4962-8c2f-1223a20c2114/kube-rbac-proxy/0.log"
Sep 09 21:09:48.340533 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.340461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zgpc8_a22e518a-a8bf-4962-8c2f-1223a20c2114/init-textfile/0.log"
Sep 09 21:09:48.754187 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.754099    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-cjx9l_3a61d7a5-ea84-4708-8fa8-8ea553254ab1/prometheus-operator/0.log"
Sep 09 21:09:48.777759 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.777731    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-cjx9l_3a61d7a5-ea84-4708-8fa8-8ea553254ab1/kube-rbac-proxy/0.log"
Sep 09 21:09:48.816478 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.816405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-75cld_8a94f10a-e0da-4cd7-9bb7-e4eda485d89a/prometheus-operator-admission-webhook/0.log"
Sep 09 21:09:48.931541 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.931482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5684db94c7-rt24t_bae5fbac-03ce-4e1a-8667-e6042edd1b5f/thanos-query/0.log"
Sep 09 21:09:48.962713 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.962676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5684db94c7-rt24t_bae5fbac-03ce-4e1a-8667-e6042edd1b5f/kube-rbac-proxy-web/0.log"
Sep 09 21:09:48.988780 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:48.988727    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5684db94c7-rt24t_bae5fbac-03ce-4e1a-8667-e6042edd1b5f/kube-rbac-proxy/0.log"
Sep 09 21:09:49.017677 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:49.017584    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5684db94c7-rt24t_bae5fbac-03ce-4e1a-8667-e6042edd1b5f/prom-label-proxy/0.log"
Sep 09 21:09:49.047332 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:49.047301    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5684db94c7-rt24t_bae5fbac-03ce-4e1a-8667-e6042edd1b5f/kube-rbac-proxy-rules/0.log"
Sep 09 21:09:49.077788 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:49.077760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5684db94c7-rt24t_bae5fbac-03ce-4e1a-8667-e6042edd1b5f/kube-rbac-proxy-metrics/0.log"
Sep 09 21:09:51.374833 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.374799    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"]
Sep 09 21:09:51.380068 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.380043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8fddf498c-tj64x_80a5acc7-2eaf-4f0d-ba74-a9648d29429d/console/0.log"
Sep 09 21:09:51.380671 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.380622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.389602 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.389173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"]
Sep 09 21:09:51.434433 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.434401    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-v7dx9_552cc6ef-0359-43f2-9e11-7706454be021/download-server/0.log"
Sep 09 21:09:51.504812 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.504778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-lib-modules\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.505345 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.505324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-sys\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.505577 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.505563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbvc\" (UniqueName: \"kubernetes.io/projected/a241da58-cbe6-4558-a99e-0a7d944b8778-kube-api-access-ctbvc\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.505971 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.505953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-proc\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.506331 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.506315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-podres\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607228 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbvc\" (UniqueName: \"kubernetes.io/projected/a241da58-cbe6-4558-a99e-0a7d944b8778-kube-api-access-ctbvc\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607417 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-proc\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607417 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-podres\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607417 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-lib-modules\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607417 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-sys\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607417 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-proc\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607625 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-podres\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607625 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-sys\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.607625 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.607492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a241da58-cbe6-4558-a99e-0a7d944b8778-lib-modules\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.620931 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.620909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbvc\" (UniqueName: \"kubernetes.io/projected/a241da58-cbe6-4558-a99e-0a7d944b8778-kube-api-access-ctbvc\") pod \"perf-node-gather-daemonset-tdrht\" (UID: \"a241da58-cbe6-4558-a99e-0a7d944b8778\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.694062 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.693978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:51.985838 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:51.985707    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"]
Sep 09 21:09:51.988616 ip-10-0-129-130 kubenswrapper[2563]: W0909 21:09:51.988582    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda241da58_cbe6_4558_a99e_0a7d944b8778.slice/crio-874fcf8e22a9a7d05823463b2d09dc76228a8b553ee4b372cfa2c3f0124c7c3e WatchSource:0}: Error finding container 874fcf8e22a9a7d05823463b2d09dc76228a8b553ee4b372cfa2c3f0124c7c3e: Status 404 returned error can't find the container with id 874fcf8e22a9a7d05823463b2d09dc76228a8b553ee4b372cfa2c3f0124c7c3e
Sep 09 21:09:52.913851 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:52.913807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht" event={"ID":"a241da58-cbe6-4558-a99e-0a7d944b8778","Type":"ContainerStarted","Data":"42e5fd06fd72eac135c576c678ba21c42bff3d8bcd96cdff0449e3035e95f417"}
Sep 09 21:09:52.914325 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:52.913910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht" event={"ID":"a241da58-cbe6-4558-a99e-0a7d944b8778","Type":"ContainerStarted","Data":"874fcf8e22a9a7d05823463b2d09dc76228a8b553ee4b372cfa2c3f0124c7c3e"}
Sep 09 21:09:52.914741 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:52.914712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:09:52.935522 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:52.935463    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht" podStartSLOduration=1.9354435479999998 podStartE2EDuration="1.935443548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:52.934403471 +0000 UTC m=+3519.973573422" watchObservedRunningTime="2026-09-09 21:09:52.935443548 +0000 UTC m=+3519.974613501"
Sep 09 21:09:53.165745 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:53.165665    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vm5kn_5df9dd2b-eb41-4eea-bab2-927f81ac0386/dns/0.log"
Sep 09 21:09:53.193297 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:53.193271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vm5kn_5df9dd2b-eb41-4eea-bab2-927f81ac0386/kube-rbac-proxy/0.log"
Sep 09 21:09:53.257036 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:53.257011    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vrpjd_d0e7c5d4-bd8e-4d59-94d9-cd43edc64cef/dns-node-resolver/0.log"
Sep 09 21:09:53.845887 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:53.845856    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rhb6f_163dd371-674d-4aa6-9c0d-11f70d0a5c80/node-ca/0.log"
Sep 09 21:09:55.085016 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:55.084987    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b69kb_da2252f6-c961-4a1d-b8b3-5923989737e9/serve-healthcheck-canary/0.log"
Sep 09 21:09:55.624434 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:55.624405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4s4k_88c270f0-4f8e-4d6f-861f-a49056c296b8/kube-rbac-proxy/0.log"
Sep 09 21:09:55.648338 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:55.648307    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4s4k_88c270f0-4f8e-4d6f-861f-a49056c296b8/exporter/0.log"
Sep 09 21:09:55.672587 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:55.672548    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4s4k_88c270f0-4f8e-4d6f-861f-a49056c296b8/extractor/0.log"
Sep 09 21:09:58.263538 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:58.263506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-55b9566966-s8fq2_6fc04e79-f455-4c36-9fcf-2db1cc5ebd4b/manager/0.log"
Sep 09 21:09:58.300721 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:58.300686    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-skkp8_ce73316c-2447-4e6b-bf1c-fd5e00579cb4/manager/0.log"
Sep 09 21:09:58.620692 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:58.620660    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-fjssq_d267f173-8274-4c7b-9b83-04c8f25f971a/manager/0.log"
Sep 09 21:09:58.924406 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:58.924298    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-q84gl_390166ac-5157-4d7c-9463-b12ea47c3e23/seaweedfs/0.log"
Sep 09 21:09:58.952998 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:58.952969    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-ljngh_6e234906-6b0e-4421-98bf-efae09647927/seaweedfs-tls-custom/0.log"
Sep 09 21:09:59.939977 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:09:59.939942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-tdrht"
Sep 09 21:10:05.043583 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.043501    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ftsv5_c8455e93-3740-42b5-b358-4d0d0df723a3/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:10:05.065361 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.065337    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ftsv5_c8455e93-3740-42b5-b358-4d0d0df723a3/egress-router-binary-copy/0.log"
Sep 09 21:10:05.085608 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.085555    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ftsv5_c8455e93-3740-42b5-b358-4d0d0df723a3/cni-plugins/0.log"
Sep 09 21:10:05.105599 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.105550    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ftsv5_c8455e93-3740-42b5-b358-4d0d0df723a3/bond-cni-plugin/0.log"
Sep 09 21:10:05.127760 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.127738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ftsv5_c8455e93-3740-42b5-b358-4d0d0df723a3/routeoverride-cni/0.log"
Sep 09 21:10:05.150329 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.150304    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ftsv5_c8455e93-3740-42b5-b358-4d0d0df723a3/whereabouts-cni-bincopy/0.log"
Sep 09 21:10:05.172876 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.172845    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ftsv5_c8455e93-3740-42b5-b358-4d0d0df723a3/whereabouts-cni/0.log"
Sep 09 21:10:05.385431 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.385398    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ntfvh_5dcc119c-416c-460b-bbec-98f4d5d44294/kube-multus/0.log"
Sep 09 21:10:05.497510 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.497477    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jklcc_a5c6873b-0047-43fc-a158-8ff4ceb8e0dd/network-metrics-daemon/0.log"
Sep 09 21:10:05.516764 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:05.516743    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jklcc_a5c6873b-0047-43fc-a158-8ff4ceb8e0dd/kube-rbac-proxy/0.log"
Sep 09 21:10:06.502727 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.502699    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/ovn-controller/0.log"
Sep 09 21:10:06.520148 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.520124    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/ovn-acl-logging/0.log"
Sep 09 21:10:06.545071 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.545037    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/kube-rbac-proxy-node/0.log"
Sep 09 21:10:06.569609 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.569580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 21:10:06.591053 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.591029    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/northd/0.log"
Sep 09 21:10:06.611753 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.611729    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/nbdb/0.log"
Sep 09 21:10:06.632992 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.632971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/sbdb/0.log"
Sep 09 21:10:06.687931 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:06.687903    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9fx7_e162fc1b-720c-4e4f-a0b6-4c0c97e6a0da/ovnkube-controller/0.log"
Sep 09 21:10:07.905898 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:07.905866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-855rx_3dcd3207-d2b6-4acb-8a11-5dffd25b0adf/network-check-target-container/0.log"
Sep 09 21:10:08.814916 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:08.814882    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g8c2v_df2e417e-a206-4060-9bed-17dea319370f/iptables-alerter/0.log"
Sep 09 21:10:09.530403 ip-10-0-129-130 kubenswrapper[2563]: I0909 21:10:09.530332    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-p9zgw_3ba1d7bf-58a0-4113-82bc-e0a5183e3f5e/tuned/0.log"