Sep 17 21:56:04.029916 ip-10-0-139-88 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 21:56:04.029926 ip-10-0-139-88 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 21:56:04.029935 ip-10-0-139-88 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 21:56:04.030228 ip-10-0-139-88 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 21:56:15.116434 ip-10-0-139-88 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 21:56:15.116451 ip-10-0-139-88 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0afc60cdfd2d420e9c71a182d8f5e9de --
Sep 17 21:58:12.041371 ip-10-0-139-88 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:58:15.534314 ip-10-0-139-88 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:15.534314 ip-10-0-139-88 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:58:15.534314 ip-10-0-139-88 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:15.534314 ip-10-0-139-88 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:58:15.534314 ip-10-0-139-88 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:15.535077 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.534386    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:58:15.537181 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537166    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:15.537181 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537180    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537184    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537187    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537190    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537193    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537196    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537199    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537202    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537205    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537207    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537210    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537213    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537216    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537219    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537221    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537224    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537226    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537229    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537232    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537234    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:15.537248 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537237    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537240    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537243    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537245    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537248    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537250    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537253    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537255    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537259    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537262    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537265    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537269    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537272    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537275    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537278    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537281    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537283    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537292    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537295    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537297    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:15.537724 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537300    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537302    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537305    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537308    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537310    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537313    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537315    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537319    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537321    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537324    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537326    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537329    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537331    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537334    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537337    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537340    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537343    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537346    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537349    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537351    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:15.538239 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537354    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537356    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537359    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537362    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537364    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537367    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537369    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537372    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537375    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537379    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537383    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537385    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537389    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537391    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537394    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537397    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537399    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537402    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537404    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537407    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:15.538717 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537409    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537412    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537415    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537419    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537422    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537425    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537429    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537432    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537435    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537438    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537440    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537443    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537445    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537448    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537451    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537454    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537457    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.537459    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538125    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538131    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:15.539201 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538134    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538137    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538140    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538148    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538151    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538154    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538156    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538159    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538161    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538164    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538167    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538169    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538172    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538175    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538177    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538179    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538182    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538185    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538189    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538191    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:15.539681 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538196    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538200    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538203    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538206    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538209    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538212    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538214    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538217    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538220    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538223    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538225    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538228    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538231    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538234    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538236    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538239    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538251    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538255    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538258    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538266    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:15.540190 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538269    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538272    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538274    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538277    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538280    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538282    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538285    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538288    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538290    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538293    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538296    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538298    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538301    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538304    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538306    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538308    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538311    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538314    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538316    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538319    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:15.540694 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538321    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538323    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538326    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538329    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538332    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538337    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538339    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538342    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538344    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538352    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538355    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538357    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538360    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538363    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538365    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538368    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538370    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538373    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538375    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:15.541195 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538378    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538380    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538383    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538386    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538388    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538391    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538394    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538396    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538399    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538401    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538404    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538406    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538409    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538411    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538414    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538416    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538419    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538421    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538864    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538870    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:15.541705 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538872    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538875    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538878    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538881    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538884    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538888    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538892    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538896    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538898    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538901    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538904    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538907    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538909    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538912    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538916    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538919    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538922    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538925    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538928    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:15.542206 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538930    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538933    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538935    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538938    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538941    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538944    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538947    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538949    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538952    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538954    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538957    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538959    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538962    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538964    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538967    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538969    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538972    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538975    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538978    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538981    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:15.542661 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538984    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538986    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538989    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538992    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538994    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538997    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.538999    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539002    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539005    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539009    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539012    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539014    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539017    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539019    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539022    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539024    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539027    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539030    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539033    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:15.543187 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539035    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539038    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539040    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539043    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539045    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539048    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539050    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539053    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539055    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539058    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539061    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539064    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539069    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539071    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539074    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539076    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539079    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539082    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539084    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539087    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:15.543646 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539089    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539092    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539094    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539097    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539099    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539116    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539119    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539122    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539125    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539128    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539131    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539133    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539136    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539139    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539141    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539144    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539146    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539149    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539151    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539212    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539244    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:58:15.544145 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539250    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539255    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539259    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539263    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539267    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539271    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539275    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539278    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539281    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539284    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539288    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539290    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539293    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539296    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539299    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539302    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539306    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539309    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539312    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539315    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539318    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539322    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539325    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:58:15.544687 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539327    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539331    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539334    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539337    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539340    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539343    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539346    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539350    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539353    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539355    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539358    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539363    2563 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539367    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539372    2563 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539375    2563 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539378    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539381    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539384    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539387    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539390    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539393    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539396    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539399    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539402    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539405    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:58:15.545255 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539407    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539410    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539413    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539416    2563 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539420    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539423    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539426    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539429    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539432    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539435    2563 flags.go:64] FLAG: --help="false"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539438    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539441    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539443    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539446    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539450    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539453    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539456    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539458    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539463    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539466    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539469    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539473    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539476    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539479    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:58:15.545903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539482    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539485    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539488    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539491    2563 flags.go:64] FLAG: --lock-file=""
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539494    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539497    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539500    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539505    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539507    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539510    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539513    2563 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539516    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539519    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539522    2563 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539525    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539529    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539536    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539545    2563 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539548    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539551    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539554    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539557    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539560    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539563    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539565    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:58:15.546520 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539572    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539576    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539580    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539583    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539586    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539592    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539595    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539598    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539601    2563 flags.go:64] FLAG: --port="10250"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539604    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539607    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bcd134db6bd4cef2"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539610    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539613    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539616    2563 flags.go:64] FLAG: --register-node="true"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539619    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539623    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539626    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539628    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539631    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539635    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539638    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539641    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539644    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539646    2563 flags.go:64] FLAG: --runonce="false"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539649    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:58:15.547161 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539652    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539655    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539658    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539661    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539664    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539667    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539670    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539672    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539675    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539681    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539684    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539688    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539691    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539694    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539700    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539702    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539705    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539709    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539712    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539715    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539718    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539721    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539724    2563 flags.go:64] FLAG: --v="2"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539728    2563 flags.go:64] FLAG: --version="false"
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539731    2563 flags.go:64] FLAG: --vmodule=""
Sep 17 21:58:15.547751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539735    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.539738    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539852    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539856    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539859    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539862    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539865    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539869    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539871    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539874    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539877    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539879    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539882    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539885    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539887    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539891    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539898    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539901    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539904    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:15.548407 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539906    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539911    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539915    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539918    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539921    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539924    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539927    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539930    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539932    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539935    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539938    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539940    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539943    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539946    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539948    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539951    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539953    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539959    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539961    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:15.548888 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539964    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539967    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539969    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539972    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539975    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539977    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539980    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539982    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539985    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539987    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539992    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539995    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.539998    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540001    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540003    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540006    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540009    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540011    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540014    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540017    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:15.549370 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540019    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540022    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540025    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540027    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540030    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540032    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540035    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540038    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540040    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540043    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540045    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540048    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540051    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540053    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540056    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540058    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540061    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540063    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540066    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:15.549857 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540068    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540071    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540073    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540079    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540081    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540084    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540087    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540090    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540092    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540095    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540098    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540113    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540117    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540120    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540122    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540125    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540127    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540130    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540133    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540136    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:15.550340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540139    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540143    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540146    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.540150    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.540155    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.547218    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.547232    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.547245    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.547250    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547304    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547310    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547313    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547316    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547319    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:15.550826 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547322    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547325    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547328    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547331    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547333    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547337    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547341    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547345    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547348    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547351    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547353    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547356    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547359    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547362    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547364    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547367    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547369    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547372    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547375    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:15.551189 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547377    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547380    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547382    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547385    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547388    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547391    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547394    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547396    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547399    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547402    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547404    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547407    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547410    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547412    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547415    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547417    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547420    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547422    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547425    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547427    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:15.551704 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547430    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547433    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547435    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547438    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547440    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547444    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547448    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547451    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547454    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547457    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547461    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547463    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547466    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547469    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547471    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547474    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547477    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547480    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547482    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:15.552329 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547485    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547488    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547490    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547493    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547495    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547498    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547500    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547503    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547505    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547508    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547512    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547516    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547518    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547521    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547523    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547526    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547529    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547531    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547534    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547536    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:15.552872 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547539    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547542    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547545    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547548    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547550    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547553    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547555    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547558    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547561    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547563    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547566    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547568    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547571    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547573    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547575    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:15.553448 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547578    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.547583    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547689    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547693    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547696    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547699    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547702    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547704    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547707    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547709    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547711    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547714    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547717    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547719    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547722    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:15.553922 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547725    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547727    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547730    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547733    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547736    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547738    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547741    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547743    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547746    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547748    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547751    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547754    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547757    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547760    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547762    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547765    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547768    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547771    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547773    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547777    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:15.554452 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547781    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547783    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547786    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547789    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547791    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547794    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547797    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547799    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547802    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547805    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547807    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547810    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547813    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547815    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547818    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547821    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547823    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547826    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547829    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:15.554947 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547831    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547833    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547836    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547839    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547841    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547843    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547846    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547849    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547851    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547854    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547856    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547858    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547861    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547864    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547866    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547869    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547871    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547874    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547876    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547879    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:15.555428 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547881    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547884    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547886    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547889    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547891    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547894    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547897    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547899    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547903    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547906    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547910    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547913    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547916    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547918    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547921    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547924    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547926    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547929    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547931    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547934    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:15.555911 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547936    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547940    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547943    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547946    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547949    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547952    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:15.547954    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.547959    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.548046    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.551039    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.551929    2563 server.go:1019] "Starting client certificate rotation"
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.552048    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.552095    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:15.556406 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.556223    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:15.557350 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.557330    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:15.566589 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.566572    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:58:15.568140 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.568125    2563 log.go:25] "Validated CRI v1 image API"
Sep 17 21:58:15.569002 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.568990    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:58:15.571174 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.571157    2563 fs.go:135] Filesystem UUIDs: map[5e36205d-a786-409c-a2a5-3e495b15379e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ded04e09-9240-4829-9fb1-a516fcce6eb6:/dev/nvme0n1p3]
Sep 17 21:58:15.571224 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.571174    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:58:15.575248 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.575154    2563 manager.go:216] Machine: {Timestamp:2026-09-17 21:58:15.574693832 +0000 UTC m=+2.058179558 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103783 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c6462fa46d8a21212f49b6297a91 SystemUUID:ec24c646-2fa4-6d8a-2121-2f49b6297a91 BootID:0afc60cd-fd2d-420e-9c71-a182d8f5e9de Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:79:d3:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:79:d3:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:36:b1:b7:52:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:58:15.575248 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.575243    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:58:15.575371 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.575310    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:58:15.577209 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.577181    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:58:15.577338 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.577215    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:58:15.578329 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.578319    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:58:15.578363 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.578331    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:58:15.578363 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.578339    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:15.578363 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.578349    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:15.578443 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.578412    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:58:15.581387 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.581376    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:15.581526 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.581517    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:58:15.581563 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.581547    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:58:15.581563 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.581556    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:58:15.581634 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.581566    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:58:15.581634 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.581575    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:58:15.581634 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.581584    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:58:15.582019 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.582001    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:15.582080 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.582035    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:15.583974 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.583959    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:58:15.587260 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.587242    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:15.589455 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.589437    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:58:15.589539 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.589476    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:58:15.590673 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590659    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590678    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590687    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590694    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590707    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590715    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590733    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590741    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:58:15.590751 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590751    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:58:15.591014 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590759    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:58:15.591014 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590777    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:58:15.591014 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590786    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:58:15.591014 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590821    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:58:15.591014 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.590830    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:58:15.593993 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.593973    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:58:15.594825 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.594804    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:58:15.594890 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.594822    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:15.595089 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.595067    2563 server.go:1268] "Started kubelet"
Sep 17 21:58:15.595201 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.595137    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:58:15.595263 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.595219    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:58:15.595317 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.595288    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:58:15.595847 ip-10-0-139-88 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:58:15.596819 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.596742    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:58:15.596875 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.596863    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:58:15.599766 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.599747    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:58:15.603272 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.603256    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:58:15.603352 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.603276    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:15.603352 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.603333    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:15.603968 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.603951    2563 factory.go:55] Registering systemd factory
Sep 17 21:58:15.603968 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.603968    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:58:15.604134 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604016    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:58:15.604134 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604022    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:58:15.604134 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604026    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:58:15.604289 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604166    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:58:15.604289 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604174    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:58:15.604289 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604248    2563 factory.go:153] Registering CRI-O factory
Sep 17 21:58:15.604289 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604264    2563 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:58:15.604447 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604332    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:58:15.604447 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604350    2563 factory.go:103] Registering Raw factory
Sep 17 21:58:15.604447 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.604364    2563 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:58:15.605345 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.605325    2563 manager.go:349] Starting recovery of all containers
Sep 17 21:58:15.606389 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.606371    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 21:58:15.606493 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.606475    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 21:58:15.616915 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.616818    2563 manager.go:354] Recovery completed
Sep 17 21:58:15.634383 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.634369    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:15.636059 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636042    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:15.636128 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636068    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:15.636128 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636079    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:15.636563 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636550    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:58:15.636563 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636561    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:58:15.636691 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636564    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:15.636691 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636579    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:15.636793 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.636778    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jbbbr"
Sep 17 21:58:15.637241 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.637230    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:15.637293 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.637255    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:15.637293 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.637264    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:15.637899 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.637888    2563 policy_none.go:49] "None policy: Start"
Sep 17 21:58:15.637957 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.637906    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:58:15.637957 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.637919    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:58:15.639062 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.639049    2563 policy_none.go:47] "Start"
Sep 17 21:58:15.641265 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.640142    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-88.ec2.internal.18d63a90d1bf7d8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-88.ec2.internal,UID:ip-10-0-139-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-88.ec2.internal,},FirstTimestamp:2026-09-17 21:58:15.595040142 +0000 UTC m=+2.078525866,LastTimestamp:2026-09-17 21:58:15.595040142 +0000 UTC m=+2.078525866,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-88.ec2.internal,}"
Sep 17 21:58:15.644041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.644026    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jbbbr"
Sep 17 21:58:15.645011 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.644945    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-88.ec2.internal.18d63a90d431644f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-88.ec2.internal,UID:ip-10-0-139-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-88.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-88.ec2.internal,},FirstTimestamp:2026-09-17 21:58:15.636059215 +0000 UTC m=+2.119544943,LastTimestamp:2026-09-17 21:58:15.636059215 +0000 UTC m=+2.119544943,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-88.ec2.internal,}"
Sep 17 21:58:15.648815 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.648739    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-88.ec2.internal.18d63a90d4319db6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-88.ec2.internal,UID:ip-10-0-139-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-88.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-88.ec2.internal,},FirstTimestamp:2026-09-17 21:58:15.63607391 +0000 UTC m=+2.119559635,LastTimestamp:2026-09-17 21:58:15.63607391 +0000 UTC m=+2.119559635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-88.ec2.internal,}"
Sep 17 21:58:15.674662 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.674648    2563 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.674687    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.674697    2563 server.go:87] "Starting device plugin registration server"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.674920    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.674945    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.675025    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.675123    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.675135    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.675543    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:58:15.683290 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.675582    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:15.703789 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.703765    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:58:15.704829 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.704811    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:58:15.704895 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.704850    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:58:15.704895 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.704870    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:58:15.704964 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.704901    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:58:15.707041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.707017    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:15.775032 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.775002    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:15.775803 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.775786    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:15.775893 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.775818    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:15.775893 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.775833    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:15.775893 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.775863    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.785900 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.785851    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.785900 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.785873    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-88.ec2.internal\": node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:15.805562 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.805541    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"]
Sep 17 21:58:15.805636 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.805597    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:15.806370 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.806346    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:15.806449 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.806378    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:15.806449 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.806390    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:15.807571 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.807559    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:15.807703 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.807689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.807744 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.807718    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:15.808275 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.808256    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:15.808275 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.808269    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:15.808357 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.808285    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:15.808357 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.808289    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:15.808357 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.808296    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:15.808357 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.808302    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:15.809407 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.809392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.809454 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.809418    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:15.810140 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.810125    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:15.810215 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.810150    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:15.810215 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.810160    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:15.815224 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.815211    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:15.841717 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.841697    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-88.ec2.internal\" not found" node="ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.846021 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.846005    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-88.ec2.internal\" not found" node="ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.905397 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.905376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cce065d5c7d1b9816d8d702b1174775-config\") pod \"kube-apiserver-proxy-ip-10-0-139-88.ec2.internal\" (UID: \"5cce065d5c7d1b9816d8d702b1174775\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.905464 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.905399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36466fc3525a31946e07a8c08667e49a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"36466fc3525a31946e07a8c08667e49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.905464 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:15.905417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36466fc3525a31946e07a8c08667e49a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"36466fc3525a31946e07a8c08667e49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:15.915460 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:15.915445    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.005914 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.005892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cce065d5c7d1b9816d8d702b1174775-config\") pod \"kube-apiserver-proxy-ip-10-0-139-88.ec2.internal\" (UID: \"5cce065d5c7d1b9816d8d702b1174775\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.006003 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.005919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36466fc3525a31946e07a8c08667e49a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"36466fc3525a31946e07a8c08667e49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.006003 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.005935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36466fc3525a31946e07a8c08667e49a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"36466fc3525a31946e07a8c08667e49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.006003 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.005982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36466fc3525a31946e07a8c08667e49a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"36466fc3525a31946e07a8c08667e49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.006145 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.005984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cce065d5c7d1b9816d8d702b1174775-config\") pod \"kube-apiserver-proxy-ip-10-0-139-88.ec2.internal\" (UID: \"5cce065d5c7d1b9816d8d702b1174775\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.006145 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.006027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36466fc3525a31946e07a8c08667e49a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"36466fc3525a31946e07a8c08667e49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.016001 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.015984    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.116680 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.116627    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.144726 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.144707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.149923 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.149907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.217407 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.217380    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.317959 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.317936    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.418414 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.418361    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.518825 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.518803    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.551981 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.551961    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:58:16.552466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.552137    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:16.573553 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.573533    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:16.619244 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.619217    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.649377 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.649339    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:53:15 +0000 UTC" deadline="2028-05-11 19:13:09.097052494 +0000 UTC"
Sep 17 21:58:16.649377 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.649373    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14445h14m52.44768221s"
Sep 17 21:58:16.703684 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.703655    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:16.704813 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.704798    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:16.704874 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.704827    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:16.704874 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.704839    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:16.720155 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.720134    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.723090 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:16.723061    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36466fc3525a31946e07a8c08667e49a.slice/crio-339ba20a455d654873eacf7eac9de4aa54d9d122a71a732ce58d337903f631da WatchSource:0}: Error finding container 339ba20a455d654873eacf7eac9de4aa54d9d122a71a732ce58d337903f631da: Status 404 returned error can't find the container with id 339ba20a455d654873eacf7eac9de4aa54d9d122a71a732ce58d337903f631da
Sep 17 21:58:16.723639 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:16.723617    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cce065d5c7d1b9816d8d702b1174775.slice/crio-79841c765d3e6a21ff22c70c3056e1a4c2282e8f338100b8973af73f57be10df WatchSource:0}: Error finding container 79841c765d3e6a21ff22c70c3056e1a4c2282e8f338100b8973af73f57be10df: Status 404 returned error can't find the container with id 79841c765d3e6a21ff22c70c3056e1a4c2282e8f338100b8973af73f57be10df
Sep 17 21:58:16.728174 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.728155    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:58:16.803642 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.803618    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:16.820598 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:16.820580    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 17 21:58:16.830912 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.830894    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:16.903475 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.903452    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.918051 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.918034    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:16.919065 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.919052    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 17 21:58:16.926673 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:16.926659    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:17.374132 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.374088    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:17.582546 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.582518    2563 apiserver.go:52] "Watching apiserver"
Sep 17 21:58:17.590049 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.589981    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:58:17.590437 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.590410    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-frql5","openshift-multus/multus-t72bk","openshift-multus/network-metrics-daemon-r22wg","openshift-network-operator/iptables-alerter-nb27h","openshift-ovn-kubernetes/ovnkube-node-776jn","kube-system/global-pull-secret-syncer-sgwj2","kube-system/konnectivity-agent-n8vq4","kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq","openshift-dns/node-resolver-284mc","openshift-image-registry/node-ca-gwsgx","openshift-network-diagnostics/network-check-target-ssgcr","openshift-cluster-node-tuning-operator/tuned-pkrfb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"]
Sep 17 21:58:17.592269 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.592245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.593568 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.593547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.594559 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.594538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:17.594653 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.594607    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:17.596386 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.595952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.596673 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.596650    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:58:17.596773 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.596687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:58:17.596846 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.596780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:58:17.596910 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.596892    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:58:17.596988 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.596971    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rnrjz\""
Sep 17 21:58:17.597040 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.597025    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:58:17.597190 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.597173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.598211 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.598193    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wg9ms\""
Sep 17 21:58:17.598460 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.598446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.599699 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.599571    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:17.599787 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.599717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:17.599787 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.599730    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:58:17.599890 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.599848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cjw6z\""
Sep 17 21:58:17.599890 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.599854    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:58:17.600098 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.600081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.600871 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.600852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m9t96\""
Sep 17 21:58:17.601201 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.601182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:58:17.601928 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.601600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.601928 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.601610    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:58:17.602297 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.602276    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:58:17.602985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.602967    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:58:17.603289 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.603269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.603980 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.603962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7b9c\""
Sep 17 21:58:17.604167 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.604150    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:58:17.604246 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.604181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:58:17.604309 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.604254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:58:17.604309 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.604263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:58:17.604408 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.604347    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:17.605088 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.605045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.605227 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.605201    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:17.605829 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.605812    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dgcjl\""
Sep 17 21:58:17.605932 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.605915    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:58:17.605987 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.605924    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:17.606065 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.606044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:58:17.606178 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.606137    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:17.606178 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.605823    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:58:17.606294 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.606266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bnq92\""
Sep 17 21:58:17.606444 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.606416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:58:17.606878 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.606861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:58:17.607247 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.607088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kqwlt\""
Sep 17 21:58:17.607247 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.607094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.610699 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.608615    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:58:17.610699 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.609902    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:58:17.611330 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.611030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:17.611330 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.611030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qvz9s\""
Sep 17 21:58:17.611497 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.611474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:17.613754 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.613734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:58:17.614123 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-iptables-alerter-script\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.614199 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614199 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24916e2a-7f9f-4375-a023-033d9af6d468-dbus\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.614199 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysconfig\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.614199 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-kubernetes\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.614395 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-var-lib-kubelet\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.614395 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:17.614395 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef78f52-2dc7-4839-ae3d-5e8519fde738-tmp-dir\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.614395 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-run\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.614395 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-kubelet\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614395 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-slash\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-systemd\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-etc-selinux\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-socket-dir-parent\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-k8s-cni-cncf-io\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966bv\" (UniqueName: \"kubernetes.io/projected/11e30628-ef09-472c-b72e-f068975056ea-kube-api-access-966bv\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hzf\" (UniqueName: \"kubernetes.io/projected/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-kube-api-access-r9hzf\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-etc-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614605 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16445f01-9a93-4ea6-885e-89db7cfdec04-ovn-node-metrics-cert\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-modprobe-d\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-socket-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-node-log\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-cnibin\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-etc-kubernetes\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-systemd\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-env-overrides\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-netns\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-daemon-config\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-multus-certs\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-var-lib-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.614967 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-system-cni-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.614984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-cni-binary-copy\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11e30628-ef09-472c-b72e-f068975056ea-serviceca\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-ovn\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxtl\" (UniqueName: \"kubernetes.io/projected/16445f01-9a93-4ea6-885e-89db7cfdec04-kube-api-access-gnxtl\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-cni-multus\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-kubelet\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-tuned\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-run-netns\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-cni-bin\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2892cc96-3ca8-42db-b1cb-363cac3b86de-agent-certs\") pod \"konnectivity-agent-n8vq4\" (UID: \"2892cc96-3ca8-42db-b1cb-363cac3b86de\") " pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-cni-netd\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-sys\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-systemd-units\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-log-socket\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-run-ovn-kubernetes\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-hostroot\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.615501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ef78f52-2dc7-4839-ae3d-5e8519fde738-hosts-file\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-lib-modules\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-cni-bin\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-conf-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2892cc96-3ca8-42db-b1cb-363cac3b86de-konnectivity-ca\") pod \"konnectivity-agent-n8vq4\" (UID: \"2892cc96-3ca8-42db-b1cb-363cac3b86de\") " pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6c6m\" (UniqueName: \"kubernetes.io/projected/7ef78f52-2dc7-4839-ae3d-5e8519fde738-kube-api-access-t6c6m\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-registration-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysctl-conf\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacb05e2-766c-4f2d-815d-204ba7bf320e-tmp\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6zm\" (UniqueName: \"kubernetes.io/projected/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-kube-api-access-xq6zm\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-ovnkube-config\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szjxq\" (UniqueName: \"kubernetes.io/projected/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-kube-api-access-szjxq\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysctl-d\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jj7\" (UniqueName: \"kubernetes.io/projected/cacb05e2-766c-4f2d-815d-204ba7bf320e-kube-api-access-k2jj7\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11e30628-ef09-472c-b72e-f068975056ea-host\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.615983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-host\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.616351 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-sys-fs\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.617219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-host-slash\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.617219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.617219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-ovnkube-script-lib\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.617219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-cni-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.617219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24916e2a-7f9f-4375-a023-033d9af6d468-kubelet-config\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.617219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.617219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.616262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-device-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.623290 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.623265    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:17.659903 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.659882    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v7n4q"
Sep 17 21:58:17.669920 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.669889    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v7n4q"
Sep 17 21:58:17.705652 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.705628    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:58:17.709519 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.709481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal" event={"ID":"5cce065d5c7d1b9816d8d702b1174775","Type":"ContainerStarted","Data":"79841c765d3e6a21ff22c70c3056e1a4c2282e8f338100b8973af73f57be10df"}
Sep 17 21:58:17.710718 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.710691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" event={"ID":"36466fc3525a31946e07a8c08667e49a","Type":"ContainerStarted","Data":"339ba20a455d654873eacf7eac9de4aa54d9d122a71a732ce58d337903f631da"}
Sep 17 21:58:17.717327 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jj7\" (UniqueName: \"kubernetes.io/projected/cacb05e2-766c-4f2d-815d-204ba7bf320e-kube-api-access-k2jj7\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.717419 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11e30628-ef09-472c-b72e-f068975056ea-host\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.717419 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-host\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.717419 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-sys-fs\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.717532 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-sys-fs\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.717532 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11e30628-ef09-472c-b72e-f068975056ea-host\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.717532 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-host\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.717663 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-host-slash\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.717663 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.717758 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-ovnkube-script-lib\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.717758 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-cni-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.717758 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-host-slash\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.717758 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.717758 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24916e2a-7f9f-4375-a023-033d9af6d468-kubelet-config\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-device-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-iptables-alerter-script\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24916e2a-7f9f-4375-a023-033d9af6d468-kubelet-config\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24916e2a-7f9f-4375-a023-033d9af6d468-dbus\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-device-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-cni-binary-copy\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-cni-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysconfig\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.717985 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.717962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24916e2a-7f9f-4375-a023-033d9af6d468-dbus\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysconfig\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-kubernetes\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-var-lib-kubelet\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef78f52-2dc7-4839-ae3d-5e8519fde738-tmp-dir\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-run\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-var-lib-kubelet\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-kubelet\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-kubernetes\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-slash\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-systemd\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-ovnkube-script-lib\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vmgg\" (UniqueName: \"kubernetes.io/projected/92cceb95-3224-4dd3-a4e2-d50bf1c58950-kube-api-access-9vmgg\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-etc-selinux\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-socket-dir-parent\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-kubelet\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.718453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-k8s-cni-cncf-io\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-k8s-cni-cncf-io\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-etc-selinux\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-slash\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966bv\" (UniqueName: \"kubernetes.io/projected/11e30628-ef09-472c-b72e-f068975056ea-kube-api-access-966bv\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-systemd\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-run\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-iptables-alerter-script\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef78f52-2dc7-4839-ae3d-5e8519fde738-tmp-dir\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-socket-dir-parent\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hzf\" (UniqueName: \"kubernetes.io/projected/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-kube-api-access-r9hzf\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.718605    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.718671    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret podName:24916e2a-7f9f-4375-a023-033d9af6d468 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:18.218649924 +0000 UTC m=+4.702135654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret") pod "global-pull-secret-syncer-sgwj2" (UID: "24916e2a-7f9f-4375-a023-033d9af6d468") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-etc-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-etc-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16445f01-9a93-4ea6-885e-89db7cfdec04-ovn-node-metrics-cert\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719081 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-cnibin\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-modprobe-d\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-socket-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-node-log\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-cnibin\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.718984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-etc-kubernetes\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-systemd\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-env-overrides\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-modprobe-d\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-netns\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-daemon-config\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-cnibin\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-multus-certs\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-systemd\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-node-log\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-etc-kubernetes\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-multus-certs\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.719755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-run-netns\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-var-lib-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-system-cni-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-cni-binary-copy\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-socket-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-var-lib-openvswitch\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11e30628-ef09-472c-b72e-f068975056ea-serviceca\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-system-cni-dir\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-system-cni-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-ovn\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxtl\" (UniqueName: \"kubernetes.io/projected/16445f01-9a93-4ea6-885e-89db7cfdec04-kube-api-access-gnxtl\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-run-ovn\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-cni-multus\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-env-overrides\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-kubelet\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-tuned\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.720466 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-cni-multus\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-run-netns\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-cni-bin\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2892cc96-3ca8-42db-b1cb-363cac3b86de-agent-certs\") pod \"konnectivity-agent-n8vq4\" (UID: \"2892cc96-3ca8-42db-b1cb-363cac3b86de\") " pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-run-netns\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-cni-netd\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-sys\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-systemd-units\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-daemon-config\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-cni-bin\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-log-socket\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-host-var-lib-kubelet\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-cni-netd\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-sys\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-log-socket\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-run-ovn-kubernetes\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-systemd-units\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11e30628-ef09-472c-b72e-f068975056ea-serviceca\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.721168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-cni-binary-copy\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-hostroot\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-run-ovn-kubernetes\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719823    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ef78f52-2dc7-4839-ae3d-5e8519fde738-hosts-file\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-lib-modules\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ef78f52-2dc7-4839-ae3d-5e8519fde738-hosts-file\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-hostroot\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-cni-bin\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-conf-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.719999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2892cc96-3ca8-42db-b1cb-363cac3b86de-konnectivity-ca\") pod \"konnectivity-agent-n8vq4\" (UID: \"2892cc96-3ca8-42db-b1cb-363cac3b86de\") " pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-lib-modules\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6c6m\" (UniqueName: \"kubernetes.io/projected/7ef78f52-2dc7-4839-ae3d-5e8519fde738-kube-api-access-t6c6m\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-multus-conf-dir\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16445f01-9a93-4ea6-885e-89db7cfdec04-host-cni-bin\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-registration-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysctl-conf\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.721964 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacb05e2-766c-4f2d-815d-204ba7bf320e-tmp\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-registration-dir\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6zm\" (UniqueName: \"kubernetes.io/projected/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-kube-api-access-xq6zm\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-ovnkube-config\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjxq\" (UniqueName: \"kubernetes.io/projected/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-kube-api-access-szjxq\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysctl-conf\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcxz\" (UniqueName: \"kubernetes.io/projected/35891657-8ef8-494c-af51-2d122db69e59-kube-api-access-sdcxz\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysctl-d\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-sysctl-d\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2892cc96-3ca8-42db-b1cb-363cac3b86de-konnectivity-ca\") pod \"konnectivity-agent-n8vq4\" (UID: \"2892cc96-3ca8-42db-b1cb-363cac3b86de\") " pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.722801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.720724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16445f01-9a93-4ea6-885e-89db7cfdec04-ovnkube-config\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.723628 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.723610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacb05e2-766c-4f2d-815d-204ba7bf320e-tmp\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.723717 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.723647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cacb05e2-766c-4f2d-815d-204ba7bf320e-etc-tuned\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.723792 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.723777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16445f01-9a93-4ea6-885e-89db7cfdec04-ovn-node-metrics-cert\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.724331 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.724314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2892cc96-3ca8-42db-b1cb-363cac3b86de-agent-certs\") pod \"konnectivity-agent-n8vq4\" (UID: \"2892cc96-3ca8-42db-b1cb-363cac3b86de\") " pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.734499 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.734478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jj7\" (UniqueName: \"kubernetes.io/projected/cacb05e2-766c-4f2d-815d-204ba7bf320e-kube-api-access-k2jj7\") pod \"tuned-pkrfb\" (UID: \"cacb05e2-766c-4f2d-815d-204ba7bf320e\") " pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.735342 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.735324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxtl\" (UniqueName: \"kubernetes.io/projected/16445f01-9a93-4ea6-885e-89db7cfdec04-kube-api-access-gnxtl\") pod \"ovnkube-node-776jn\" (UID: \"16445f01-9a93-4ea6-885e-89db7cfdec04\") " pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.736045 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.736028    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:17.736132 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.736050    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:17.736132 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.736063    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k2whx for pod openshift-network-diagnostics/network-check-target-ssgcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:17.736206 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.736136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx podName:a1797ce8-ce06-4cb4-acbc-f253f4d85ef7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:18.236118513 +0000 UTC m=+4.719604228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k2whx" (UniqueName: "kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx") pod "network-check-target-ssgcr" (UID: "a1797ce8-ce06-4cb4-acbc-f253f4d85ef7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:17.739241 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.739219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966bv\" (UniqueName: \"kubernetes.io/projected/11e30628-ef09-472c-b72e-f068975056ea-kube-api-access-966bv\") pod \"node-ca-gwsgx\" (UID: \"11e30628-ef09-472c-b72e-f068975056ea\") " pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.739988 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.739940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjxq\" (UniqueName: \"kubernetes.io/projected/7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6-kube-api-access-szjxq\") pod \"multus-t72bk\" (UID: \"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6\") " pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.740122 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.740085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6c6m\" (UniqueName: \"kubernetes.io/projected/7ef78f52-2dc7-4839-ae3d-5e8519fde738-kube-api-access-t6c6m\") pod \"node-resolver-284mc\" (UID: \"7ef78f52-2dc7-4839-ae3d-5e8519fde738\") " pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.740674 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.740653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hzf\" (UniqueName: \"kubernetes.io/projected/f09458f3-a0b3-4662-bf01-84e80a8ebcd6-kube-api-access-r9hzf\") pod \"iptables-alerter-nb27h\" (UID: \"f09458f3-a0b3-4662-bf01-84e80a8ebcd6\") " pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.740782 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.740766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6zm\" (UniqueName: \"kubernetes.io/projected/e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f-kube-api-access-xq6zm\") pod \"aws-ebs-csi-driver-node-66kxq\" (UID: \"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.821538 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:17.821695 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.821695 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcxz\" (UniqueName: \"kubernetes.io/projected/35891657-8ef8-494c-af51-2d122db69e59-kube-api-access-sdcxz\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.821695 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-cni-binary-copy\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.821695 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.821594    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:17.821695 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vmgg\" (UniqueName: \"kubernetes.io/projected/92cceb95-3224-4dd3-a4e2-d50bf1c58950-kube-api-access-9vmgg\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:17.821695 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:17.821651    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:18.321633494 +0000 UTC m=+4.805119210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:17.821695 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-cnibin\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-system-cni-dir\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-cnibin\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-system-cni-dir\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.821995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35891657-8ef8-494c-af51-2d122db69e59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822272 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.822191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822272 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.822212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-cni-binary-copy\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.822362 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.822311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35891657-8ef8-494c-af51-2d122db69e59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.832671 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.832644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vmgg\" (UniqueName: \"kubernetes.io/projected/92cceb95-3224-4dd3-a4e2-d50bf1c58950-kube-api-access-9vmgg\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:17.833281 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.833259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcxz\" (UniqueName: \"kubernetes.io/projected/35891657-8ef8-494c-af51-2d122db69e59-kube-api-access-sdcxz\") pod \"multus-additional-cni-plugins-frql5\" (UID: \"35891657-8ef8-494c-af51-2d122db69e59\") " pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:17.908065 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.907994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:17.914150 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.914131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq"
Sep 17 21:58:17.925172 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.925154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nb27h"
Sep 17 21:58:17.931618 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.931602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gwsgx"
Sep 17 21:58:17.938170 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.938150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-284mc"
Sep 17 21:58:17.945828 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.945812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:17.953980 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.953963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pkrfb"
Sep 17 21:58:17.960089 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.960071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t72bk"
Sep 17 21:58:17.967086 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:17.967065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frql5"
Sep 17 21:58:18.224239 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.224171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:18.224369 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.224303    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:18.224415 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.224368    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret podName:24916e2a-7f9f-4375-a023-033d9af6d468 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:19.224349518 +0000 UTC m=+5.707835233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret") pod "global-pull-secret-syncer-sgwj2" (UID: "24916e2a-7f9f-4375-a023-033d9af6d468") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:18.325398 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.325364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:18.325549 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.325420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:18.325549 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.325521    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:18.325549 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.325548    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:18.325679 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.325561    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:18.325679 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.325573    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k2whx for pod openshift-network-diagnostics/network-check-target-ssgcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:18.325679 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.325588    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:19.325571083 +0000 UTC m=+5.809056808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:18.325679 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:18.325608    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx podName:a1797ce8-ce06-4cb4-acbc-f253f4d85ef7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:19.325595686 +0000 UTC m=+5.809081407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2whx" (UniqueName: "kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx") pod "network-check-target-ssgcr" (UID: "a1797ce8-ce06-4cb4-acbc-f253f4d85ef7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:18.518823 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.518792    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacb05e2_766c_4f2d_815d_204ba7bf320e.slice/crio-ce21538fc5499102ebdd3a0066abb6d4fe8b9cdf7132bfbc07d83f5a127c23c6 WatchSource:0}: Error finding container ce21538fc5499102ebdd3a0066abb6d4fe8b9cdf7132bfbc07d83f5a127c23c6: Status 404 returned error can't find the container with id ce21538fc5499102ebdd3a0066abb6d4fe8b9cdf7132bfbc07d83f5a127c23c6
Sep 17 21:58:18.521339 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.521170    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2892cc96_3ca8_42db_b1cb_363cac3b86de.slice/crio-6c0a5309efd4141ee5371e5e059ab16512f807066e61bac0e37f53d9b89de4db WatchSource:0}: Error finding container 6c0a5309efd4141ee5371e5e059ab16512f807066e61bac0e37f53d9b89de4db: Status 404 returned error can't find the container with id 6c0a5309efd4141ee5371e5e059ab16512f807066e61bac0e37f53d9b89de4db
Sep 17 21:58:18.524679 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.524652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35891657_8ef8_494c_af51_2d122db69e59.slice/crio-92988a7bb0acbb880ab326f219c9e9dab873172a4c00446314c9024187b8fd1f WatchSource:0}: Error finding container 92988a7bb0acbb880ab326f219c9e9dab873172a4c00446314c9024187b8fd1f: Status 404 returned error can't find the container with id 92988a7bb0acbb880ab326f219c9e9dab873172a4c00446314c9024187b8fd1f
Sep 17 21:58:18.525840 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.525813    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f50b9a_d51e_4f03_baf9_8d5a9b7f789f.slice/crio-dc3e47e8466e091fc9694adeb7c57c8f69cdb02b9e11891f58544cdd5514e8fb WatchSource:0}: Error finding container dc3e47e8466e091fc9694adeb7c57c8f69cdb02b9e11891f58544cdd5514e8fb: Status 404 returned error can't find the container with id dc3e47e8466e091fc9694adeb7c57c8f69cdb02b9e11891f58544cdd5514e8fb
Sep 17 21:58:18.526650 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.526633    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e30628_ef09_472c_b72e_f068975056ea.slice/crio-b3fb4125063c0571eaceff46225ef817cb1a856f7858d0e0386514d6c127813e WatchSource:0}: Error finding container b3fb4125063c0571eaceff46225ef817cb1a856f7858d0e0386514d6c127813e: Status 404 returned error can't find the container with id b3fb4125063c0571eaceff46225ef817cb1a856f7858d0e0386514d6c127813e
Sep 17 21:58:18.527247 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.527210    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef78f52_2dc7_4839_ae3d_5e8519fde738.slice/crio-fb2588b53bc2a0421192a81c44d7a851f357c89b16052ca08eda22c3557b2488 WatchSource:0}: Error finding container fb2588b53bc2a0421192a81c44d7a851f357c89b16052ca08eda22c3557b2488: Status 404 returned error can't find the container with id fb2588b53bc2a0421192a81c44d7a851f357c89b16052ca08eda22c3557b2488
Sep 17 21:58:18.529494 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.528439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16445f01_9a93_4ea6_885e_89db7cfdec04.slice/crio-0a9e2a40e9daa9d6a6e33ef05b56108f8aef2df1b3155b298448d8058ad52141 WatchSource:0}: Error finding container 0a9e2a40e9daa9d6a6e33ef05b56108f8aef2df1b3155b298448d8058ad52141: Status 404 returned error can't find the container with id 0a9e2a40e9daa9d6a6e33ef05b56108f8aef2df1b3155b298448d8058ad52141
Sep 17 21:58:18.529776 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.529731    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09458f3_a0b3_4662_bf01_84e80a8ebcd6.slice/crio-345f1aab06cf5bfb5fac8a326cfb2b002480feffff1d72f3331076a2b64433a4 WatchSource:0}: Error finding container 345f1aab06cf5bfb5fac8a326cfb2b002480feffff1d72f3331076a2b64433a4: Status 404 returned error can't find the container with id 345f1aab06cf5bfb5fac8a326cfb2b002480feffff1d72f3331076a2b64433a4
Sep 17 21:58:18.531498 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:18.531473    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2c9e9f_9b3a_41c5_9f64_d39e3426a6f6.slice/crio-291a0879db22709fc64c77f6768e6f6bbc7d951719691ebfbfefcb7ef310ea2c WatchSource:0}: Error finding container 291a0879db22709fc64c77f6768e6f6bbc7d951719691ebfbfefcb7ef310ea2c: Status 404 returned error can't find the container with id 291a0879db22709fc64c77f6768e6f6bbc7d951719691ebfbfefcb7ef310ea2c
Sep 17 21:58:18.672711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.672667    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:17 +0000 UTC" deadline="2028-06-18 00:39:22.681409016 +0000 UTC"
Sep 17 21:58:18.672711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.672710    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15338h41m4.008703212s"
Sep 17 21:58:18.714452 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.714426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"0a9e2a40e9daa9d6a6e33ef05b56108f8aef2df1b3155b298448d8058ad52141"}
Sep 17 21:58:18.716720 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.716699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerStarted","Data":"92988a7bb0acbb880ab326f219c9e9dab873172a4c00446314c9024187b8fd1f"}
Sep 17 21:58:18.719010 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.718988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal" event={"ID":"5cce065d5c7d1b9816d8d702b1174775","Type":"ContainerStarted","Data":"bf3b99cbec2aff47a369057f110595d9578c081eafa96cb476e226485280bc45"}
Sep 17 21:58:18.720489 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.720463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t72bk" event={"ID":"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6","Type":"ContainerStarted","Data":"291a0879db22709fc64c77f6768e6f6bbc7d951719691ebfbfefcb7ef310ea2c"}
Sep 17 21:58:18.721600 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.721574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nb27h" event={"ID":"f09458f3-a0b3-4662-bf01-84e80a8ebcd6","Type":"ContainerStarted","Data":"345f1aab06cf5bfb5fac8a326cfb2b002480feffff1d72f3331076a2b64433a4"}
Sep 17 21:58:18.722769 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.722672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-284mc" event={"ID":"7ef78f52-2dc7-4839-ae3d-5e8519fde738","Type":"ContainerStarted","Data":"fb2588b53bc2a0421192a81c44d7a851f357c89b16052ca08eda22c3557b2488"}
Sep 17 21:58:18.723684 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.723664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gwsgx" event={"ID":"11e30628-ef09-472c-b72e-f068975056ea","Type":"ContainerStarted","Data":"b3fb4125063c0571eaceff46225ef817cb1a856f7858d0e0386514d6c127813e"}
Sep 17 21:58:18.724720 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.724699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq" event={"ID":"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f","Type":"ContainerStarted","Data":"dc3e47e8466e091fc9694adeb7c57c8f69cdb02b9e11891f58544cdd5514e8fb"}
Sep 17 21:58:18.725761 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.725741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8vq4" event={"ID":"2892cc96-3ca8-42db-b1cb-363cac3b86de","Type":"ContainerStarted","Data":"6c0a5309efd4141ee5371e5e059ab16512f807066e61bac0e37f53d9b89de4db"}
Sep 17 21:58:18.726779 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.726757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pkrfb" event={"ID":"cacb05e2-766c-4f2d-815d-204ba7bf320e","Type":"ContainerStarted","Data":"ce21538fc5499102ebdd3a0066abb6d4fe8b9cdf7132bfbc07d83f5a127c23c6"}
Sep 17 21:58:18.734556 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:18.734512    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal" podStartSLOduration=2.734501236 podStartE2EDuration="2.734501236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:18.734234378 +0000 UTC m=+5.217720114" watchObservedRunningTime="2026-09-17 21:58:18.734501236 +0000 UTC m=+5.217986969"
Sep 17 21:58:19.232901 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.232526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:19.232901 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.232707    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:19.232901 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.232767    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret podName:24916e2a-7f9f-4375-a023-033d9af6d468 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:21.232750005 +0000 UTC m=+7.716235720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret") pod "global-pull-secret-syncer-sgwj2" (UID: "24916e2a-7f9f-4375-a023-033d9af6d468") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:19.333637 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.333597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:19.333816 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.333665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:19.333880 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.333833    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:19.333880 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.333855    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:19.333880 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.333867    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k2whx for pod openshift-network-diagnostics/network-check-target-ssgcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:19.334040 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.333923    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx podName:a1797ce8-ce06-4cb4-acbc-f253f4d85ef7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:21.333905318 +0000 UTC m=+7.817391046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2whx" (UniqueName: "kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx") pod "network-check-target-ssgcr" (UID: "a1797ce8-ce06-4cb4-acbc-f253f4d85ef7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:19.334355 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.334316    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:19.334428 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.334370    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:21.334353797 +0000 UTC m=+7.817839515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:19.673197 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.673077    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:17 +0000 UTC" deadline="2028-05-07 01:56:41.670720606 +0000 UTC"
Sep 17 21:58:19.673197 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.673149    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14331h58m21.997576053s"
Sep 17 21:58:19.706298 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.706234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:19.706446 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.706237    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:19.706446 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.706365    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:19.706446 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.706429    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:19.706608 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.706497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:19.706608 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:19.706570    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:19.741209 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.741087    2563 generic.go:356] "Generic (PLEG): container finished" podID="36466fc3525a31946e07a8c08667e49a" containerID="8e6a38ef7c9c5f3f4de0ee70728dbfd3be311044cec072061d5e952b5931b716" exitCode=0
Sep 17 21:58:19.741341 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:19.741146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" event={"ID":"36466fc3525a31946e07a8c08667e49a","Type":"ContainerDied","Data":"8e6a38ef7c9c5f3f4de0ee70728dbfd3be311044cec072061d5e952b5931b716"}
Sep 17 21:58:20.756270 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:20.756235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" event={"ID":"36466fc3525a31946e07a8c08667e49a","Type":"ContainerStarted","Data":"a1c2eeadf5ddc8c10416640680892b80b0de103151726428f3a77b7280548331"}
Sep 17 21:58:21.250043 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:21.249962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:21.250220 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.250162    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:21.250285 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.250225    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret podName:24916e2a-7f9f-4375-a023-033d9af6d468 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.250208444 +0000 UTC m=+11.733694161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret") pod "global-pull-secret-syncer-sgwj2" (UID: "24916e2a-7f9f-4375-a023-033d9af6d468") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:21.350710 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:21.350668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:21.350882 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:21.350742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:21.350882 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.350821    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:21.350989 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.350883    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.350866275 +0000 UTC m=+11.834351998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:21.350989 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.350917    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:21.350989 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.350968    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:21.350989 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.350981    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k2whx for pod openshift-network-diagnostics/network-check-target-ssgcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:21.351255 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.351023    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx podName:a1797ce8-ce06-4cb4-acbc-f253f4d85ef7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.351010818 +0000 UTC m=+11.834496544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2whx" (UniqueName: "kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx") pod "network-check-target-ssgcr" (UID: "a1797ce8-ce06-4cb4-acbc-f253f4d85ef7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:21.705446 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:21.705348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:21.705613 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.705471    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:21.705613 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:21.705524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:21.705739 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.705612    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:21.705739 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:21.705643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:21.705837 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:21.705766    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:23.705196 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:23.705157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:23.705648 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:23.705282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:23.705724 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:23.705646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:23.705773 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:23.705749    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:23.705908 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:23.705886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:23.705991 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:23.705971    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:25.281115 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:25.281058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:25.281534 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.281221    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.281534 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.281291    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret podName:24916e2a-7f9f-4375-a023-033d9af6d468 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:33.281271385 +0000 UTC m=+19.764757100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret") pod "global-pull-secret-syncer-sgwj2" (UID: "24916e2a-7f9f-4375-a023-033d9af6d468") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.381612 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:25.381576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:25.381778 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:25.381644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:25.381778 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.381745    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.381778 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.381760    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:25.381778 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.381778    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:25.381960 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.381791    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k2whx for pod openshift-network-diagnostics/network-check-target-ssgcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:25.381960 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.381817    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:33.381799053 +0000 UTC m=+19.865284783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.381960 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.381835    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx podName:a1797ce8-ce06-4cb4-acbc-f253f4d85ef7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:33.381826057 +0000 UTC m=+19.865311772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2whx" (UniqueName: "kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx") pod "network-check-target-ssgcr" (UID: "a1797ce8-ce06-4cb4-acbc-f253f4d85ef7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:25.706254 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:25.706184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:25.706395 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.706293    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:25.707675 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:25.707650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:25.707799 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.707738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:25.707921 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:25.707903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:25.708031 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:25.708011    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:27.705476 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:27.705312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:27.705476 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:27.705334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:27.705476 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:27.705454    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:27.705932 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:27.705551    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:27.705932 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:27.705581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:27.705932 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:27.705624    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:29.705805 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:29.705768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:29.705805 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:29.705808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:29.706370 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:29.705768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:29.706370 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:29.705908    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:29.706370 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:29.705990    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:29.706370 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:29.706121    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:31.706118 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:31.706073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:31.706538 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:31.706125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:31.706538 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:31.706079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:31.706538 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:31.706210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:31.706538 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:31.706298    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:31.706538 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:31.706376    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:33.344674 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:33.344637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:33.345224 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.344762    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:33.345224 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.344828    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret podName:24916e2a-7f9f-4375-a023-033d9af6d468 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:49.344807926 +0000 UTC m=+35.828293639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret") pod "global-pull-secret-syncer-sgwj2" (UID: "24916e2a-7f9f-4375-a023-033d9af6d468") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:33.445789 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:33.445747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:33.445965 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:33.445815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:33.446029 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.445956    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:33.446029 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.445983    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:33.446029 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.445996    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k2whx for pod openshift-network-diagnostics/network-check-target-ssgcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:33.446029 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.445997    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:33.446259 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.446063    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:49.446041582 +0000 UTC m=+35.929527309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:33.446259 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.446082    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx podName:a1797ce8-ce06-4cb4-acbc-f253f4d85ef7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:49.446072703 +0000 UTC m=+35.929558420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2whx" (UniqueName: "kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx") pod "network-check-target-ssgcr" (UID: "a1797ce8-ce06-4cb4-acbc-f253f4d85ef7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:33.705204 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:33.705118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:33.705371 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:33.705127    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:33.705371 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.705240    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:33.705371 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:33.705258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:33.705371 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.705341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:33.705584 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:33.705406    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:35.706545 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:35.706488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:35.706926 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:35.706571    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:35.706926 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:35.706676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:35.706926 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:35.706808    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:35.706926 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:35.706843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:35.706926 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:35.706917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:36.789186 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.788751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq" event={"ID":"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f","Type":"ContainerStarted","Data":"f0cd846d96e23a08a89cd54e3ffe84f3a448c79f842fe2fb9cd760ca079d0e98"}
Sep 17 21:58:36.790565 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.790496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8vq4" event={"ID":"2892cc96-3ca8-42db-b1cb-363cac3b86de","Type":"ContainerStarted","Data":"e8ceb4f8e85be564f93e300c4064367d4b261c5ca95bf48d276c41390c706b5c"}
Sep 17 21:58:36.792223 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.792193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pkrfb" event={"ID":"cacb05e2-766c-4f2d-815d-204ba7bf320e","Type":"ContainerStarted","Data":"989e5a2bcaad88a45a3c49b0ad3b0970062102b61d109d72317b60117a8ab324"}
Sep 17 21:58:36.795596 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.795579    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 21:58:36.795969 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.795949    2563 generic.go:356] "Generic (PLEG): container finished" podID="16445f01-9a93-4ea6-885e-89db7cfdec04" containerID="16d4a4e82da3e12f0425ec65f5f79d083e3c1062d030cab341c87c15a1998282" exitCode=1
Sep 17 21:58:36.796078 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.796014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"2d9f574edeb2ae2ee0b13521c45d90ab6ad3566b69089192937a683650923386"}
Sep 17 21:58:36.796145 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.796090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"e19a21087a1945203973e8d01291df384666933dd033fbb5478552d482367579"}
Sep 17 21:58:36.796192 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.796165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"ae4d589e66c0b7ce2287870f19605ba0287b7db22e25f5e5ac2b122bac3f9255"}
Sep 17 21:58:36.796232 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.796214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"ffe2b542aa33245f2b14a86e9b852c7f61ed7dd156c94e89c75088eb00939dd0"}
Sep 17 21:58:36.796277 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.796262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerDied","Data":"16d4a4e82da3e12f0425ec65f5f79d083e3c1062d030cab341c87c15a1998282"}
Sep 17 21:58:36.796337 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.796307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"158ee0cb9f9a92813ec6e0c1eb06bbce1f2b6c22d4798b285ec1533248af63e0"}
Sep 17 21:58:36.797796 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.797775    2563 generic.go:356] "Generic (PLEG): container finished" podID="35891657-8ef8-494c-af51-2d122db69e59" containerID="0d9a1ef2dcc9bded763b643941bdc2448d5117cd8de147ff77a71747298a7648" exitCode=0
Sep 17 21:58:36.797884 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.797798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerDied","Data":"0d9a1ef2dcc9bded763b643941bdc2448d5117cd8de147ff77a71747298a7648"}
Sep 17 21:58:36.799530 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.799501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t72bk" event={"ID":"7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6","Type":"ContainerStarted","Data":"24f588276aa787f6def2c4b5a14ac59065308a2061d2bca31f2d04a67402647d"}
Sep 17 21:58:36.801442 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.801413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-284mc" event={"ID":"7ef78f52-2dc7-4839-ae3d-5e8519fde738","Type":"ContainerStarted","Data":"db79d91ec1823bd121b8eb85bb503e976292638998e97327ed2752ed3ea89428"}
Sep 17 21:58:36.803091 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.803068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gwsgx" event={"ID":"11e30628-ef09-472c-b72e-f068975056ea","Type":"ContainerStarted","Data":"a7896d8372f6a06f7e157a7c5306a8a7c93e89614bf685caeb14c585f976347e"}
Sep 17 21:58:36.804730 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.804688    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" podStartSLOduration=20.804674965 podStartE2EDuration="20.804674965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:20.787633995 +0000 UTC m=+7.271119729" watchObservedRunningTime="2026-09-17 21:58:36.804674965 +0000 UTC m=+23.288160904"
Sep 17 21:58:36.805241 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.805213    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n8vq4" podStartSLOduration=4.878810364 podStartE2EDuration="21.805204372s" totalImagesPullingTime="16.926394008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:36.804637118 +0000 UTC m=+23.288122857" watchObservedRunningTime="2026-09-17 21:58:36.805204372 +0000 UTC m=+23.288690106"
Sep 17 21:58:36.838308 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.838269    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t72bk" podStartSLOduration=3.450233529 podStartE2EDuration="20.838256599s" totalImagesPullingTime="17.38802307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:36.819584929 +0000 UTC m=+23.303070663" watchObservedRunningTime="2026-09-17 21:58:36.838256599 +0000 UTC m=+23.321742332"
Sep 17 21:58:36.851334 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.851294    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gwsgx" podStartSLOduration=4.549359576 podStartE2EDuration="21.851282329s" totalImagesPullingTime="17.301922753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:36.850882248 +0000 UTC m=+23.334367981" watchObservedRunningTime="2026-09-17 21:58:36.851282329 +0000 UTC m=+23.334768064"
Sep 17 21:58:36.886647 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.886593    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-284mc" podStartSLOduration=4.50499593 podStartE2EDuration="21.886575384s" totalImagesPullingTime="17.381579454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:36.864100389 +0000 UTC m=+23.347586124" watchObservedRunningTime="2026-09-17 21:58:36.886575384 +0000 UTC m=+23.370061100"
Sep 17 21:58:36.886836 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:36.886810    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pkrfb" podStartSLOduration=3.496109063 podStartE2EDuration="20.886800345s" totalImagesPullingTime="17.390691282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:36.886524584 +0000 UTC m=+23.370010321" watchObservedRunningTime="2026-09-17 21:58:36.886800345 +0000 UTC m=+23.370286078"
Sep 17 21:58:37.298788 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.298766    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:58:37.688362 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.688231    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:58:37.298783394Z","UUID":"122ad43b-3e6a-460e-b0c8-474d23ff581d","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:58:37.689911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.689890    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:58:37.690034 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.689922    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:58:37.705623 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.705598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:37.705770 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.705625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:37.705770 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.705694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:37.705770 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:37.705720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:37.705925 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:37.705837    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:37.705982 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:37.705939    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:37.806949 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.806914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nb27h" event={"ID":"f09458f3-a0b3-4662-bf01-84e80a8ebcd6","Type":"ContainerStarted","Data":"a312f74514ab7905a16f7072aa12cc85551dca94c72c7b13c151631524567d94"}
Sep 17 21:58:37.809301 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:37.809272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq" event={"ID":"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f","Type":"ContainerStarted","Data":"9328fffae351a046c8bf1eff3cf4311bd482509b212ec9ac8ccb223cc92eb22b"}
Sep 17 21:58:38.816001 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:38.815969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq" event={"ID":"e3f50b9a-d51e-4f03-baf9-8d5a9b7f789f","Type":"ContainerStarted","Data":"79958cd803f9670e440fe60ab60751161852e6adaf43954b05bd7a227d517da5"}
Sep 17 21:58:38.833836 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:38.833787    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nb27h" podStartSLOduration=6.455008282 podStartE2EDuration="23.833770621s" totalImagesPullingTime="17.378762339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:37.82192674 +0000 UTC m=+24.305412476" watchObservedRunningTime="2026-09-17 21:58:38.833770621 +0000 UTC m=+25.317256354"
Sep 17 21:58:38.834148 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:38.834117    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66kxq" podStartSLOduration=4.449376489 podStartE2EDuration="23.834094811s" totalImagesPullingTime="19.384718322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:38.833472207 +0000 UTC m=+25.316957941" watchObservedRunningTime="2026-09-17 21:58:38.834094811 +0000 UTC m=+25.317580547"
Sep 17 21:58:39.705780 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:39.705548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:39.705975 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:39.705548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:39.705975 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:39.705884    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:39.705975 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:39.705625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:39.706165 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:39.705982    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:39.706165 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:39.706067    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:39.821280 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:39.821249    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 21:58:39.821711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:39.821688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"534d76cf96f0cbf9f3c9dc40ad9f72def8521098998438616455e8ca499833df"}
Sep 17 21:58:41.114536 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.114498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:41.115237 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.115204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:41.705677 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.705645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:41.705808 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.705676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:41.705808 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.705645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:41.705808 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:41.705755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:41.705908 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:41.705814    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:41.705940 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:41.705894    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:41.829388 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.829361    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 21:58:41.829752 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.829733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"882e0b0dc4c49a1145dbd03262f30c81794268f01ba7fb2c0a4c4486cc9313e6"}
Sep 17 21:58:41.829952 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.829928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:41.830174 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.830158    2563 scope.go:117] "RemoveContainer" containerID="16d4a4e82da3e12f0425ec65f5f79d083e3c1062d030cab341c87c15a1998282"
Sep 17 21:58:41.831726 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.831706    2563 generic.go:356] "Generic (PLEG): container finished" podID="35891657-8ef8-494c-af51-2d122db69e59" containerID="ffa0cc8de24cf9368c55223f064f547d55bef4bd3362cd5de7dd09d05afebd0c" exitCode=0
Sep 17 21:58:41.831830 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.831797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerDied","Data":"ffa0cc8de24cf9368c55223f064f547d55bef4bd3362cd5de7dd09d05afebd0c"}
Sep 17 21:58:41.846243 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:41.846224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:42.838201 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.838172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 21:58:42.838601 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.838553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" event={"ID":"16445f01-9a93-4ea6-885e-89db7cfdec04","Type":"ContainerStarted","Data":"d6594e4896931de22d4b524d7c70a5a738b4b6fafcac65dfebec94bc2ca1ab01"}
Sep 17 21:58:42.838700 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.838686    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:42.838958 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.838928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:42.854380 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.854357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:42.868812 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.868479    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" podStartSLOduration=9.445573881 podStartE2EDuration="26.86846051s" totalImagesPullingTime="17.422886629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:42.866895715 +0000 UTC m=+29.350381449" watchObservedRunningTime="2026-09-17 21:58:42.86846051 +0000 UTC m=+29.351946236"
Sep 17 21:58:42.885431 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.884883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sgwj2"]
Sep 17 21:58:42.885431 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.884998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:42.885431 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:42.885099    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:42.885719 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.885696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r22wg"]
Sep 17 21:58:42.885822 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.885800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:42.885926 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:42.885896    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:42.886448 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.886418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssgcr"]
Sep 17 21:58:42.886560 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:42.886544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:42.886672 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:42.886646    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:43.842146 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:43.842093    2563 generic.go:356] "Generic (PLEG): container finished" podID="35891657-8ef8-494c-af51-2d122db69e59" containerID="8d6b8606902889fdd63365fb5c7da95ee6cc99b417db9d6114298ff13b0ba7e7" exitCode=0
Sep 17 21:58:43.842481 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:43.842191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerDied","Data":"8d6b8606902889fdd63365fb5c7da95ee6cc99b417db9d6114298ff13b0ba7e7"}
Sep 17 21:58:43.842481 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:43.842324    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:44.705801 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:44.705771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:44.705975 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:44.705771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:44.705975 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:44.705869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:44.705975 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:44.705772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:44.706129 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:44.705960    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:44.706129 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:44.706063    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:44.843783 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:44.843760    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:44.846326 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:44.846309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:58:45.493394 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:45.493344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:45.493543 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:45.493456    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:45.494019 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:45.493997    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n8vq4"
Sep 17 21:58:45.850127 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:45.850034    2563 generic.go:356] "Generic (PLEG): container finished" podID="35891657-8ef8-494c-af51-2d122db69e59" containerID="1d09707a5e3366629cc81aecc84fe2f85b9f533ec2cb26cf09325176dcae0503" exitCode=0
Sep 17 21:58:45.850465 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:45.850134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerDied","Data":"1d09707a5e3366629cc81aecc84fe2f85b9f533ec2cb26cf09325176dcae0503"}
Sep 17 21:58:45.865639 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:45.865610    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-776jn" podUID="16445f01-9a93-4ea6-885e-89db7cfdec04" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 21:58:46.705477 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:46.705439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:46.705645 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:46.705448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:46.705645 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:46.705569    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:46.705645 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:46.705584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:46.705825 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:46.705699    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:46.705825 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:46.705779    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:48.705763 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.705731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:48.706266 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.705778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:48.706266 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.705740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:48.706266 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:48.705851    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgwj2" podUID="24916e2a-7f9f-4375-a023-033d9af6d468"
Sep 17 21:58:48.706266 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:48.705952    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssgcr" podUID="a1797ce8-ce06-4cb4-acbc-f253f4d85ef7"
Sep 17 21:58:48.706266 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:48.706036    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r22wg" podUID="92cceb95-3224-4dd3-a4e2-d50bf1c58950"
Sep 17 21:58:48.938537 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.937004    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeReady"
Sep 17 21:58:48.938537 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.937183    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:58:48.975271 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.975238    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fdc6496f7-k2zhg"]
Sep 17 21:58:48.997188 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.997010    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x8hln"]
Sep 17 21:58:48.997188 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.997157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:48.999680 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.999655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 21:58:48.999906 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:48.999891    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 21:58:49.000136 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.000095    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hqrds\""
Sep 17 21:58:49.000250 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.000153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 21:58:49.005857 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.005834    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 21:58:49.011357 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.011335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fdc6496f7-k2zhg"]
Sep 17 21:58:49.011460 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.011374    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-slrhp"]
Sep 17 21:58:49.011460 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.011450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.013981 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.013958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb92b\""
Sep 17 21:58:49.014080 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.014026    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:58:49.014080 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.014068    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:58:49.032367 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.032286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slrhp"]
Sep 17 21:58:49.032367 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.032324    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x8hln"]
Sep 17 21:58:49.032531 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.032403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.034928 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.034905    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:58:49.034928 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.034924    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hxw4c\""
Sep 17 21:58:49.035073 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.034960    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:58:49.035143 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.035078    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:58:49.166453 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.166643 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-trusted-ca\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.166643 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgmks\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-kube-api-access-kgmks\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.166643 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjl7k\" (UniqueName: \"kubernetes.io/projected/34304a54-4fe0-4cbd-888a-d65272625bbf-kube-api-access-xjl7k\") pod \"ingress-canary-slrhp\" (UID: \"34304a54-4fe0-4cbd-888a-d65272625bbf\") " pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.166643 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-certificates\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.166643 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-installation-pull-secrets\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.166890 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-ca-trust-extracted\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.166890 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7x8x\" (UniqueName: \"kubernetes.io/projected/a214a81a-0083-4cf0-8226-294fcc318e37-kube-api-access-w7x8x\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.166890 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.166890 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-bound-sa-token\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.166890 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a214a81a-0083-4cf0-8226-294fcc318e37-config-volume\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.167151 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214a81a-0083-4cf0-8226-294fcc318e37-tmp-dir\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.167151 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34304a54-4fe0-4cbd-888a-d65272625bbf-cert\") pod \"ingress-canary-slrhp\" (UID: \"34304a54-4fe0-4cbd-888a-d65272625bbf\") " pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.167151 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.166975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-image-registry-private-configuration\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.267727 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214a81a-0083-4cf0-8226-294fcc318e37-tmp-dir\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34304a54-4fe0-4cbd-888a-d65272625bbf-cert\") pod \"ingress-canary-slrhp\" (UID: \"34304a54-4fe0-4cbd-888a-d65272625bbf\") " pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-image-registry-private-configuration\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-trusted-ca\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmks\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-kube-api-access-kgmks\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjl7k\" (UniqueName: \"kubernetes.io/projected/34304a54-4fe0-4cbd-888a-d65272625bbf-kube-api-access-xjl7k\") pod \"ingress-canary-slrhp\" (UID: \"34304a54-4fe0-4cbd-888a-d65272625bbf\") " pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-certificates\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.267911 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-installation-pull-secrets\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-ca-trust-extracted\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7x8x\" (UniqueName: \"kubernetes.io/projected/a214a81a-0083-4cf0-8226-294fcc318e37-kube-api-access-w7x8x\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.267962    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.267984    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fdc6496f7-k2zhg: secret "image-registry-tls" not found
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.267993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.268018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-bound-sa-token\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.268042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a214a81a-0083-4cf0-8226-294fcc318e37-config-volume\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.268078    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls podName:bee37e1e-49fd-42ae-bcbd-942a1a2898e7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:49.76805753 +0000 UTC m=+36.251543262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls") pod "image-registry-7fdc6496f7-k2zhg" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7") : secret "image-registry-tls" not found
Sep 17 21:58:49.268316 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.268091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214a81a-0083-4cf0-8226-294fcc318e37-tmp-dir\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.268753 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.268471    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:49.268753 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.268528    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls podName:a214a81a-0083-4cf0-8226-294fcc318e37 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:49.768512097 +0000 UTC m=+36.251997824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls") pod "dns-default-x8hln" (UID: "a214a81a-0083-4cf0-8226-294fcc318e37") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:49.268753 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.268578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a214a81a-0083-4cf0-8226-294fcc318e37-config-volume\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.268753 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.268468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-ca-trust-extracted\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.269009 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.268985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-certificates\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.269218 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.269180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-trusted-ca\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.272245 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.272223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-installation-pull-secrets\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.272346 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.272305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-image-registry-private-configuration\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.272395 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.272347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34304a54-4fe0-4cbd-888a-d65272625bbf-cert\") pod \"ingress-canary-slrhp\" (UID: \"34304a54-4fe0-4cbd-888a-d65272625bbf\") " pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.277138 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.277055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgmks\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-kube-api-access-kgmks\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.277506 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.277482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjl7k\" (UniqueName: \"kubernetes.io/projected/34304a54-4fe0-4cbd-888a-d65272625bbf-kube-api-access-xjl7k\") pod \"ingress-canary-slrhp\" (UID: \"34304a54-4fe0-4cbd-888a-d65272625bbf\") " pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.277861 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.277837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7x8x\" (UniqueName: \"kubernetes.io/projected/a214a81a-0083-4cf0-8226-294fcc318e37-kube-api-access-w7x8x\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.278017 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.277992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-bound-sa-token\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.343131 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.343084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slrhp"
Sep 17 21:58:49.368845 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.368606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:49.368845 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.368774    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:49.368942 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.368859    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret podName:24916e2a-7f9f-4375-a023-033d9af6d468 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.368836916 +0000 UTC m=+67.852322649 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret") pod "global-pull-secret-syncer-sgwj2" (UID: "24916e2a-7f9f-4375-a023-033d9af6d468") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:49.469262 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.469228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:49.469438 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.469328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:49.469438 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.469421    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:49.469438 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.469423    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:49.469584 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.469449    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:49.469584 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.469463    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k2whx for pod openshift-network-diagnostics/network-check-target-ssgcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:49.469584 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.469488    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.469472639 +0000 UTC m=+67.952958562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:49.469584 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.469515    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx podName:a1797ce8-ce06-4cb4-acbc-f253f4d85ef7 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.469498803 +0000 UTC m=+67.952984529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2whx" (UniqueName: "kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx") pod "network-check-target-ssgcr" (UID: "a1797ce8-ce06-4cb4-acbc-f253f4d85ef7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:49.771549 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.771521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:49.772035 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:49.771585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:49.772035 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.771673    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:49.772035 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.771674    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:49.772035 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.771739    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls podName:a214a81a-0083-4cf0-8226-294fcc318e37 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:50.771720369 +0000 UTC m=+37.255206086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls") pod "dns-default-x8hln" (UID: "a214a81a-0083-4cf0-8226-294fcc318e37") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:49.772035 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.771684    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fdc6496f7-k2zhg: secret "image-registry-tls" not found
Sep 17 21:58:49.772035 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:49.771796    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls podName:bee37e1e-49fd-42ae-bcbd-942a1a2898e7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:50.771785669 +0000 UTC m=+37.255271381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls") pod "image-registry-7fdc6496f7-k2zhg" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7") : secret "image-registry-tls" not found
Sep 17 21:58:50.038296 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.038230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slrhp"]
Sep 17 21:58:50.041779 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:58:50.041754    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34304a54_4fe0_4cbd_888a_d65272625bbf.slice/crio-f77516f4f8a3a3110a51e361b1a09d343c0b53746633ed28b93682d4ecad743a WatchSource:0}: Error finding container f77516f4f8a3a3110a51e361b1a09d343c0b53746633ed28b93682d4ecad743a: Status 404 returned error can't find the container with id f77516f4f8a3a3110a51e361b1a09d343c0b53746633ed28b93682d4ecad743a
Sep 17 21:58:50.706150 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.705928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:58:50.706381 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.705927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:58:50.706381 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.705974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:58:50.709614 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.709589    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:58:50.709741 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.709663    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:58:50.710632 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.710611    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:58:50.710765 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.710749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzq2x\""
Sep 17 21:58:50.710872 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.710854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd56j\""
Sep 17 21:58:50.710943 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.710882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:58:50.778932 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.778914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:50.779436 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.778968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:50.779436 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:50.779068    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:50.779436 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:50.779091    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:50.779436 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:50.779114    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fdc6496f7-k2zhg: secret "image-registry-tls" not found
Sep 17 21:58:50.779436 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:50.779155    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls podName:a214a81a-0083-4cf0-8226-294fcc318e37 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.77913509 +0000 UTC m=+39.262620817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls") pod "dns-default-x8hln" (UID: "a214a81a-0083-4cf0-8226-294fcc318e37") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:50.779436 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:50.779174    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls podName:bee37e1e-49fd-42ae-bcbd-942a1a2898e7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.779164679 +0000 UTC m=+39.262650397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls") pod "image-registry-7fdc6496f7-k2zhg" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7") : secret "image-registry-tls" not found
Sep 17 21:58:50.865498 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.865467    2563 generic.go:356] "Generic (PLEG): container finished" podID="35891657-8ef8-494c-af51-2d122db69e59" containerID="5a8962772876251d488a458111b061b6b14f8807cfe53ee6b63d5dd4067610de" exitCode=0
Sep 17 21:58:50.865655 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.865562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerDied","Data":"5a8962772876251d488a458111b061b6b14f8807cfe53ee6b63d5dd4067610de"}
Sep 17 21:58:50.867494 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:50.867467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slrhp" event={"ID":"34304a54-4fe0-4cbd-888a-d65272625bbf","Type":"ContainerStarted","Data":"f77516f4f8a3a3110a51e361b1a09d343c0b53746633ed28b93682d4ecad743a"}
Sep 17 21:58:51.873957 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:51.873910    2563 generic.go:356] "Generic (PLEG): container finished" podID="35891657-8ef8-494c-af51-2d122db69e59" containerID="88de90c42be3dd0209cefbd42ef6b7f44f8b10e8a61dfa782f6a2aafc6dfc0c3" exitCode=0
Sep 17 21:58:51.874448 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:51.873981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerDied","Data":"88de90c42be3dd0209cefbd42ef6b7f44f8b10e8a61dfa782f6a2aafc6dfc0c3"}
Sep 17 21:58:52.794540 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:52.794507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:52.794742 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:52.794586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:52.794742 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:52.794701    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:52.794742 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:52.794721    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.794886 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:52.794787    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls podName:a214a81a-0083-4cf0-8226-294fcc318e37 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:56.794766209 +0000 UTC m=+43.278251942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls") pod "dns-default-x8hln" (UID: "a214a81a-0083-4cf0-8226-294fcc318e37") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.794886 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:52.794724    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fdc6496f7-k2zhg: secret "image-registry-tls" not found
Sep 17 21:58:52.794886 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:52.794878    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls podName:bee37e1e-49fd-42ae-bcbd-942a1a2898e7 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:56.794860558 +0000 UTC m=+43.278346284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls") pod "image-registry-7fdc6496f7-k2zhg" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7") : secret "image-registry-tls" not found
Sep 17 21:58:52.881971 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:52.881937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frql5" event={"ID":"35891657-8ef8-494c-af51-2d122db69e59","Type":"ContainerStarted","Data":"0c7b24a0a8c15f797a9a42b9b79b694e35d3de1e7fa1c2e31c722c490b09a503"}
Sep 17 21:58:52.883358 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:52.883339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slrhp" event={"ID":"34304a54-4fe0-4cbd-888a-d65272625bbf","Type":"ContainerStarted","Data":"75497efef19422ead7d688322acbd4545e601a6965b72f74e22ae2a67e459224"}
Sep 17 21:58:52.905944 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:52.905899    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-frql5" podStartSLOduration=8.628566431 podStartE2EDuration="36.905885812s" totalImagesPullingTime="28.277319381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:52.904296809 +0000 UTC m=+39.387782544" watchObservedRunningTime="2026-09-17 21:58:52.905885812 +0000 UTC m=+39.389371546"
Sep 17 21:58:52.920310 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:52.920265    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-slrhp" podStartSLOduration=2.647550187 podStartE2EDuration="4.920249315s" totalImagesPullingTime="2.272699128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:52.919655748 +0000 UTC m=+39.403141483" watchObservedRunningTime="2026-09-17 21:58:52.920249315 +0000 UTC m=+39.403735050"
Sep 17 21:58:56.824813 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:56.824778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:58:56.825203 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:58:56.824830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:58:56.825203 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:56.824916    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:56.825203 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:56.824920    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:56.825203 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:56.824938    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fdc6496f7-k2zhg: secret "image-registry-tls" not found
Sep 17 21:58:56.825203 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:56.824962    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls podName:a214a81a-0083-4cf0-8226-294fcc318e37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:04.824949732 +0000 UTC m=+51.308435444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls") pod "dns-default-x8hln" (UID: "a214a81a-0083-4cf0-8226-294fcc318e37") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:56.825203 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:58:56.824980    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls podName:bee37e1e-49fd-42ae-bcbd-942a1a2898e7 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:04.824967815 +0000 UTC m=+51.308453528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls") pod "image-registry-7fdc6496f7-k2zhg" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7") : secret "image-registry-tls" not found
Sep 17 21:59:04.882930 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:04.882884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:59:04.883324 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:04.882948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:59:04.883324 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:04.883054    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:04.883324 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:04.883065    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:04.883324 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:04.883088    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fdc6496f7-k2zhg: secret "image-registry-tls" not found
Sep 17 21:59:04.883324 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:04.883124    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls podName:a214a81a-0083-4cf0-8226-294fcc318e37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.883091703 +0000 UTC m=+67.366577415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls") pod "dns-default-x8hln" (UID: "a214a81a-0083-4cf0-8226-294fcc318e37") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:04.883324 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:04.883171    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls podName:bee37e1e-49fd-42ae-bcbd-942a1a2898e7 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.883153251 +0000 UTC m=+67.366638969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls") pod "image-registry-7fdc6496f7-k2zhg" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7") : secret "image-registry-tls" not found
Sep 17 21:59:15.858950 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:15.858916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-776jn"
Sep 17 21:59:16.424365 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.424335    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"]
Sep 17 21:59:16.427860 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.427845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.430517 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.430492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 21:59:16.430636 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.430535    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ckpzk\""
Sep 17 21:59:16.431349 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.431332    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.431434 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.431380    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.431434 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.431397    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 21:59:16.440580 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.440559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"]
Sep 17 21:59:16.461524 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.461498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-config\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.461610 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.461539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-serving-cert\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.461610 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.461561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgldh\" (UniqueName: \"kubernetes.io/projected/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-kube-api-access-zgldh\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.562315 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.562289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-config\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.562455 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.562344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-serving-cert\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.562455 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.562383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgldh\" (UniqueName: \"kubernetes.io/projected/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-kube-api-access-zgldh\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.562803 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.562781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-config\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.565835 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.565818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-serving-cert\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.570600 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.570582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgldh\" (UniqueName: \"kubernetes.io/projected/264fbc4a-e424-4eff-9d4c-0ba1e486a36e-kube-api-access-zgldh\") pod \"service-ca-operator-57cdc4b768-4ffmz\" (UID: \"264fbc4a-e424-4eff-9d4c-0ba1e486a36e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:16.737507 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:16.737475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"
Sep 17 21:59:17.004805 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:17.004729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz"]
Sep 17 21:59:17.008593 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:17.008562    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264fbc4a_e424_4eff_9d4c_0ba1e486a36e.slice/crio-e0578f7c64f05fbeb1601b112f6643ae6203a1a465b7e1d4d7726310495d5689 WatchSource:0}: Error finding container e0578f7c64f05fbeb1601b112f6643ae6203a1a465b7e1d4d7726310495d5689: Status 404 returned error can't find the container with id e0578f7c64f05fbeb1601b112f6643ae6203a1a465b7e1d4d7726310495d5689
Sep 17 21:59:17.945280 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:17.945244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz" event={"ID":"264fbc4a-e424-4eff-9d4c-0ba1e486a36e","Type":"ContainerStarted","Data":"e0578f7c64f05fbeb1601b112f6643ae6203a1a465b7e1d4d7726310495d5689"}
Sep 17 21:59:18.949305 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:18.949263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz" event={"ID":"264fbc4a-e424-4eff-9d4c-0ba1e486a36e","Type":"ContainerStarted","Data":"17806305da0f27dfefb6f6e81bbddcc0206ce620b26400d03d1c97f81597249b"}
Sep 17 21:59:18.967315 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:18.967267    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz" podStartSLOduration=1.273746407 podStartE2EDuration="2.967253347s" totalImagesPullingTime="1.69350694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:18.966304309 +0000 UTC m=+65.449790048" watchObservedRunningTime="2026-09-17 21:59:18.967253347 +0000 UTC m=+65.450739075"
Sep 17 21:59:20.861756 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.861720    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-24tcl"]
Sep 17 21:59:20.863793 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.863774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.870544 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.870527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:59:20.870889 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.870874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:59:20.871239 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.871225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:59:20.872226 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.872211    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wm66n\""
Sep 17 21:59:20.885183 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.885164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:59:20.891074 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.891056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24tcl"]
Sep 17 21:59:20.895036 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.895017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:59:20.895140 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.895063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.895140 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.895090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.895250 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.895139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:59:20.895250 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.895159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpv8\" (UniqueName: \"kubernetes.io/projected/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-api-access-2lpv8\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.895250 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.895176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/971a845b-3d43-4ca0-8ad5-c6863b6b497a-crio-socket\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.895250 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.895193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/971a845b-3d43-4ca0-8ad5-c6863b6b497a-data-volume\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.895250 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.895201    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:20.895479 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.895250    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:20.895479 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.895268    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls podName:a214a81a-0083-4cf0-8226-294fcc318e37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:52.895246684 +0000 UTC m=+99.378732415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls") pod "dns-default-x8hln" (UID: "a214a81a-0083-4cf0-8226-294fcc318e37") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:20.895479 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.895270    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fdc6496f7-k2zhg: secret "image-registry-tls" not found
Sep 17 21:59:20.895479 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.895340    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls podName:bee37e1e-49fd-42ae-bcbd-942a1a2898e7 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:52.895324604 +0000 UTC m=+99.378810321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls") pod "image-registry-7fdc6496f7-k2zhg" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7") : secret "image-registry-tls" not found
Sep 17 21:59:20.995628 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.995601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.995755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.995640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.995755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.995689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpv8\" (UniqueName: \"kubernetes.io/projected/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-api-access-2lpv8\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.995755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.995707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/971a845b-3d43-4ca0-8ad5-c6863b6b497a-crio-socket\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.995755 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.995724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/971a845b-3d43-4ca0-8ad5-c6863b6b497a-data-volume\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.995755 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.995734    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.995950 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.995784    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.495768745 +0000 UTC m=+67.979254457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.995950 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.995783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/971a845b-3d43-4ca0-8ad5-c6863b6b497a-crio-socket\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:20.995950 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.995865    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.995950 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:20.995904    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.495896745 +0000 UTC m=+67.979382488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.996079 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:20.996035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/971a845b-3d43-4ca0-8ad5-c6863b6b497a-data-volume\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:21.008268 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.008247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpv8\" (UniqueName: \"kubernetes.io/projected/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-api-access-2lpv8\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:21.399360 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.399330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:59:21.402062 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.402045    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:21.412403 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.412378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24916e2a-7f9f-4375-a023-033d9af6d468-original-pull-secret\") pod \"global-pull-secret-syncer-sgwj2\" (UID: \"24916e2a-7f9f-4375-a023-033d9af6d468\") " pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:59:21.500057 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.500021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 21:59:21.500219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.500088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:59:21.500219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.500137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:21.500219 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.500168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:21.500385 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:21.500250    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:21.500385 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:21.500261    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:21.500385 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:21.500299    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.500285937 +0000 UTC m=+68.983771662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:21.500385 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:21.500326    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.500308302 +0000 UTC m=+68.983794038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:21.503233 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.503217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:59:21.503281 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.503241    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:21.510242 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:21.510226    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 21:59:21.510298 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:21.510269    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs podName:92cceb95-3224-4dd3-a4e2-d50bf1c58950 nodeName:}" failed. No retries permitted until 2026-09-17 22:00:25.510256734 +0000 UTC m=+131.993742446 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs") pod "network-metrics-daemon-r22wg" (UID: "92cceb95-3224-4dd3-a4e2-d50bf1c58950") : secret "metrics-daemon-secret" not found
Sep 17 21:59:21.512431 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.512412    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:59:21.523192 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.523173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2whx\" (UniqueName: \"kubernetes.io/projected/a1797ce8-ce06-4cb4-acbc-f253f4d85ef7-kube-api-access-k2whx\") pod \"network-check-target-ssgcr\" (UID: \"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7\") " pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:59:21.617928 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.617905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgwj2"
Sep 17 21:59:21.626650 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.626633    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd56j\""
Sep 17 21:59:21.635608 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.635589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:59:21.895518 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.895495    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sgwj2"]
Sep 17 21:59:21.896645 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:21.896615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24916e2a_7f9f_4375_a023_033d9af6d468.slice/crio-4e59e2380df933e017bd04f029e1cc316ce68b249520d45d38d0580e1f21460a WatchSource:0}: Error finding container 4e59e2380df933e017bd04f029e1cc316ce68b249520d45d38d0580e1f21460a: Status 404 returned error can't find the container with id 4e59e2380df933e017bd04f029e1cc316ce68b249520d45d38d0580e1f21460a
Sep 17 21:59:21.912565 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.912508    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssgcr"]
Sep 17 21:59:21.915521 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:21.915501    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1797ce8_ce06_4cb4_acbc_f253f4d85ef7.slice/crio-3e5e3c8fc02d4d485f4ae5d6056ac60e8d4a8d0b0e67b4f42990ba74671c543a WatchSource:0}: Error finding container 3e5e3c8fc02d4d485f4ae5d6056ac60e8d4a8d0b0e67b4f42990ba74671c543a: Status 404 returned error can't find the container with id 3e5e3c8fc02d4d485f4ae5d6056ac60e8d4a8d0b0e67b4f42990ba74671c543a
Sep 17 21:59:21.957355 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.957331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssgcr" event={"ID":"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7","Type":"ContainerStarted","Data":"3e5e3c8fc02d4d485f4ae5d6056ac60e8d4a8d0b0e67b4f42990ba74671c543a"}
Sep 17 21:59:21.958329 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:21.958304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sgwj2" event={"ID":"24916e2a-7f9f-4375-a023-033d9af6d468","Type":"ContainerStarted","Data":"4e59e2380df933e017bd04f029e1cc316ce68b249520d45d38d0580e1f21460a"}
Sep 17 21:59:22.134629 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.134605    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"]
Sep 17 21:59:22.137209 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.137193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"
Sep 17 21:59:22.140158 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.140133    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:22.141239 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.141218    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n4wxw\""
Sep 17 21:59:22.141335 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.141249    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 21:59:22.153837 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.153817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"]
Sep 17 21:59:22.205664 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.205641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhlh\" (UniqueName: \"kubernetes.io/projected/341e77d8-ba73-4550-898e-db1c0b2e6faf-kube-api-access-sdhlh\") pod \"migrator-76794cbd6c-k4lqk\" (UID: \"341e77d8-ba73-4550-898e-db1c0b2e6faf\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"
Sep 17 21:59:22.306152 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.306122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhlh\" (UniqueName: \"kubernetes.io/projected/341e77d8-ba73-4550-898e-db1c0b2e6faf-kube-api-access-sdhlh\") pod \"migrator-76794cbd6c-k4lqk\" (UID: \"341e77d8-ba73-4550-898e-db1c0b2e6faf\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"
Sep 17 21:59:22.315621 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.315598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhlh\" (UniqueName: \"kubernetes.io/projected/341e77d8-ba73-4550-898e-db1c0b2e6faf-kube-api-access-sdhlh\") pod \"migrator-76794cbd6c-k4lqk\" (UID: \"341e77d8-ba73-4550-898e-db1c0b2e6faf\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"
Sep 17 21:59:22.446717 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.446678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"
Sep 17 21:59:22.507426 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.507305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:22.507426 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.507361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:22.507559 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:22.507549    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.507770 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:22.507615    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.507595725 +0000 UTC m=+70.991081457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.508146 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:22.508085    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.508251 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:22.508154    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.508139222 +0000 UTC m=+70.991624938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.581216 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.581191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-284mc_7ef78f52-2dc7-4839-ae3d-5e8519fde738/dns-node-resolver/0.log"
Sep 17 21:59:22.732171 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.732088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk"]
Sep 17 21:59:22.736127 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:22.736079    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341e77d8_ba73_4550_898e_db1c0b2e6faf.slice/crio-438604db1c5483bc0cb94e838cde030d83fc722ebcbd414a2986785c55ba4e3c WatchSource:0}: Error finding container 438604db1c5483bc0cb94e838cde030d83fc722ebcbd414a2986785c55ba4e3c: Status 404 returned error can't find the container with id 438604db1c5483bc0cb94e838cde030d83fc722ebcbd414a2986785c55ba4e3c
Sep 17 21:59:22.962663 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:22.962625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk" event={"ID":"341e77d8-ba73-4550-898e-db1c0b2e6faf","Type":"ContainerStarted","Data":"438604db1c5483bc0cb94e838cde030d83fc722ebcbd414a2986785c55ba4e3c"}
Sep 17 21:59:23.381123 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:23.381012    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gwsgx_11e30628-ef09-472c-b72e-f068975056ea/node-ca/0.log"
Sep 17 21:59:23.966720 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:23.966641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk" event={"ID":"341e77d8-ba73-4550-898e-db1c0b2e6faf","Type":"ContainerStarted","Data":"248064a9f7e769340f39b0a2324df28abbd26c3877b0be19e9b26b4560aaa256"}
Sep 17 21:59:24.379871 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:24.379831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-slrhp_34304a54-4fe0-4cbd-888a-d65272625bbf/serve-healthcheck-canary/0.log"
Sep 17 21:59:24.526870 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:24.526830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:24.527066 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:24.526888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:24.527066 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:24.527004    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:24.527066 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:24.527012    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:24.527066 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:24.527069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.527054452 +0000 UTC m=+75.010540164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:24.527282 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:24.527085    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.527076719 +0000 UTC m=+75.010562431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:26.975863 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:26.975827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk" event={"ID":"341e77d8-ba73-4550-898e-db1c0b2e6faf","Type":"ContainerStarted","Data":"32f94107e07cba88af4c92c26a1d9db146ec4a7ce053f6f712c4fe4fb2937710"}
Sep 17 21:59:26.977432 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:26.977409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssgcr" event={"ID":"a1797ce8-ce06-4cb4-acbc-f253f4d85ef7","Type":"ContainerStarted","Data":"5de77a63b9dc627c602f0b4ec9f5fc875f8d95271c0af0acd321e87a9b52cbf8"}
Sep 17 21:59:26.977501 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:26.977458    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:59:26.978838 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:26.978812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sgwj2" event={"ID":"24916e2a-7f9f-4375-a023-033d9af6d468","Type":"ContainerStarted","Data":"2d96bfee02dadfdb0da0b9a09bb5e8cec79ea65a0a9e41380966442836e8d896"}
Sep 17 21:59:27.006771 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:27.006729    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k4lqk" podStartSLOduration=3.884727779 podStartE2EDuration="5.006716668s" totalImagesPullingTime="1.121988889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:26.992378391 +0000 UTC m=+73.475864125" watchObservedRunningTime="2026-09-17 21:59:27.006716668 +0000 UTC m=+73.490202401"
Sep 17 21:59:27.007171 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:27.007123    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sgwj2" podStartSLOduration=66.613014878 podStartE2EDuration="1m11.007097986s" totalImagesPullingTime="4.394083108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:27.006457455 +0000 UTC m=+73.489943191" watchObservedRunningTime="2026-09-17 21:59:27.007097986 +0000 UTC m=+73.490583719"
Sep 17 21:59:27.022251 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:27.022208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ssgcr" podStartSLOduration=67.653692671 podStartE2EDuration="1m12.022197644s" totalImagesPullingTime="4.368504973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:27.021229815 +0000 UTC m=+73.504715548" watchObservedRunningTime="2026-09-17 21:59:27.022197644 +0000 UTC m=+73.505683378"
Sep 17 21:59:28.557701 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:28.557587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:28.557701 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:28.557645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:28.557701 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:28.557603    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:28.558123 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:28.557715    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:28.558123 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:28.557745    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:36.557731083 +0000 UTC m=+83.041216796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:28.558123 ip-10-0-139-88 kubenswrapper[2563]: E0917 21:59:28.557759    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm podName:971a845b-3d43-4ca0-8ad5-c6863b6b497a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:36.557752581 +0000 UTC m=+83.041238292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24tcl" (UID: "971a845b-3d43-4ca0-8ad5-c6863b6b497a") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:36.609763 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:36.609723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:36.609763 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:36.609764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:36.610340 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:36.610318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/971a845b-3d43-4ca0-8ad5-c6863b6b497a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:36.612041 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:36.612019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/971a845b-3d43-4ca0-8ad5-c6863b6b497a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24tcl\" (UID: \"971a845b-3d43-4ca0-8ad5-c6863b6b497a\") " pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:36.772293 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:36.772261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24tcl"
Sep 17 21:59:37.040701 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:37.040544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24tcl"]
Sep 17 21:59:37.043464 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:37.043436    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971a845b_3d43_4ca0_8ad5_c6863b6b497a.slice/crio-be5561af5824444b342cb57ca677de0e5c5213e2c25f6f144d31b3645582ea33 WatchSource:0}: Error finding container be5561af5824444b342cb57ca677de0e5c5213e2c25f6f144d31b3645582ea33: Status 404 returned error can't find the container with id be5561af5824444b342cb57ca677de0e5c5213e2c25f6f144d31b3645582ea33
Sep 17 21:59:38.011440 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:38.011412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24tcl" event={"ID":"971a845b-3d43-4ca0-8ad5-c6863b6b497a","Type":"ContainerStarted","Data":"086884532f86ddb31fa354d730ecbf429ba5c000a17f438ef2eebb03c5ae7db5"}
Sep 17 21:59:38.011711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:38.011469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24tcl" event={"ID":"971a845b-3d43-4ca0-8ad5-c6863b6b497a","Type":"ContainerStarted","Data":"be5561af5824444b342cb57ca677de0e5c5213e2c25f6f144d31b3645582ea33"}
Sep 17 21:59:39.016996 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:39.016950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24tcl" event={"ID":"971a845b-3d43-4ca0-8ad5-c6863b6b497a","Type":"ContainerStarted","Data":"abba3114de54341d9b7e58217a661d7309d0c788255094ac49e61badd12fc62a"}
Sep 17 21:59:40.022167 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:40.022131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24tcl" event={"ID":"971a845b-3d43-4ca0-8ad5-c6863b6b497a","Type":"ContainerStarted","Data":"1bc41c7a3eed987cf88795387b11bf8b8e76d3e565439842f54e1d6afacfd791"}
Sep 17 21:59:40.052383 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:40.052335    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-24tcl" podStartSLOduration=17.842674862 podStartE2EDuration="20.052319177s" totalImagesPullingTime="2.209644315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:40.051457314 +0000 UTC m=+86.534943040" watchObservedRunningTime="2026-09-17 21:59:40.052319177 +0000 UTC m=+86.535804943"
Sep 17 21:59:45.910165 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.910124    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"]
Sep 17 21:59:45.912422 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.912406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:45.915792 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.915754    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:59:45.915950 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.915936    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-thkmm\""
Sep 17 21:59:45.916694 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.916681    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:59:45.916978 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.916962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 21:59:45.917059 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.916994    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:59:45.932068 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.932043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"]
Sep 17 21:59:45.983534 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.983510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rs6\" (UniqueName: \"kubernetes.io/projected/eabdedc8-7b65-4b22-a838-42a79385bbec-kube-api-access-w8rs6\") pod \"managed-serviceaccount-addon-agent-559958847d-qfwtq\" (UID: \"eabdedc8-7b65-4b22-a838-42a79385bbec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:45.983668 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:45.983542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eabdedc8-7b65-4b22-a838-42a79385bbec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-559958847d-qfwtq\" (UID: \"eabdedc8-7b65-4b22-a838-42a79385bbec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:46.059237 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.059206    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-7bvxc"]
Sep 17 21:59:46.061349 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.061329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-7bvxc"
Sep 17 21:59:46.064131 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.064099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-p2vx8\""
Sep 17 21:59:46.068031 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.064879    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 21:59:46.068031 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.065206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 21:59:46.079020 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.078998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-7bvxc"]
Sep 17 21:59:46.084185 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.084165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eabdedc8-7b65-4b22-a838-42a79385bbec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-559958847d-qfwtq\" (UID: \"eabdedc8-7b65-4b22-a838-42a79385bbec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:46.084293 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.084280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rs6\" (UniqueName: \"kubernetes.io/projected/eabdedc8-7b65-4b22-a838-42a79385bbec-kube-api-access-w8rs6\") pod \"managed-serviceaccount-addon-agent-559958847d-qfwtq\" (UID: \"eabdedc8-7b65-4b22-a838-42a79385bbec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:46.086479 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.086461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eabdedc8-7b65-4b22-a838-42a79385bbec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-559958847d-qfwtq\" (UID: \"eabdedc8-7b65-4b22-a838-42a79385bbec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:46.104332 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.104308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rs6\" (UniqueName: \"kubernetes.io/projected/eabdedc8-7b65-4b22-a838-42a79385bbec-kube-api-access-w8rs6\") pod \"managed-serviceaccount-addon-agent-559958847d-qfwtq\" (UID: \"eabdedc8-7b65-4b22-a838-42a79385bbec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:46.184632 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.184548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmsv\" (UniqueName: \"kubernetes.io/projected/a90ed78c-1eae-416d-8d2f-f4195c1c6b08-kube-api-access-jtmsv\") pod \"downloads-6594f4c75-7bvxc\" (UID: \"a90ed78c-1eae-416d-8d2f-f4195c1c6b08\") " pod="openshift-console/downloads-6594f4c75-7bvxc"
Sep 17 21:59:46.235367 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.235329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"
Sep 17 21:59:46.285648 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.285611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmsv\" (UniqueName: \"kubernetes.io/projected/a90ed78c-1eae-416d-8d2f-f4195c1c6b08-kube-api-access-jtmsv\") pod \"downloads-6594f4c75-7bvxc\" (UID: \"a90ed78c-1eae-416d-8d2f-f4195c1c6b08\") " pod="openshift-console/downloads-6594f4c75-7bvxc"
Sep 17 21:59:46.297612 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.297592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmsv\" (UniqueName: \"kubernetes.io/projected/a90ed78c-1eae-416d-8d2f-f4195c1c6b08-kube-api-access-jtmsv\") pod \"downloads-6594f4c75-7bvxc\" (UID: \"a90ed78c-1eae-416d-8d2f-f4195c1c6b08\") " pod="openshift-console/downloads-6594f4c75-7bvxc"
Sep 17 21:59:46.373410 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.373376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-7bvxc"
Sep 17 21:59:46.423230 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.423207    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67dbddc5b4-5sp9t"]
Sep 17 21:59:46.425619 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.425599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.428203 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.428172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5m25g\""
Sep 17 21:59:46.428203 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.428184    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 21:59:46.428354 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.428235    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 21:59:46.428411 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.428354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 21:59:46.428618 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.428589    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 21:59:46.428741 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.428700    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 21:59:46.438443 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.438400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67dbddc5b4-5sp9t"]
Sep 17 21:59:46.486540 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.486509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-service-ca\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.486540 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.486537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-oauth-config\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.486711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.486558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t78s\" (UniqueName: \"kubernetes.io/projected/8ca563cf-0666-4ad1-8fce-b65ce76011de-kube-api-access-2t78s\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.486711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.486679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-oauth-serving-cert\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.486711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.486704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-serving-cert\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.486863 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.486737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-config\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.507140 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.507100    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq"]
Sep 17 21:59:46.509918 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:46.509897    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeabdedc8_7b65_4b22_a838_42a79385bbec.slice/crio-8855c61b1cf93085143117df712b11565a97b8afd9f325b34dbd5224c4241b17 WatchSource:0}: Error finding container 8855c61b1cf93085143117df712b11565a97b8afd9f325b34dbd5224c4241b17: Status 404 returned error can't find the container with id 8855c61b1cf93085143117df712b11565a97b8afd9f325b34dbd5224c4241b17
Sep 17 21:59:46.588030 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-oauth-serving-cert\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588158 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-serving-cert\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588158 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-config\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588158 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-service-ca\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588317 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-oauth-config\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588317 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t78s\" (UniqueName: \"kubernetes.io/projected/8ca563cf-0666-4ad1-8fce-b65ce76011de-kube-api-access-2t78s\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588769 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-service-ca\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588869 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-oauth-serving-cert\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.588869 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.588812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-config\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.590399 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.590375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-serving-cert\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.590474 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.590419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-oauth-config\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.599044 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.599026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t78s\" (UniqueName: \"kubernetes.io/projected/8ca563cf-0666-4ad1-8fce-b65ce76011de-kube-api-access-2t78s\") pod \"console-67dbddc5b4-5sp9t\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") " pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:46.641232 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.641210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-7bvxc"]
Sep 17 21:59:46.644234 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:46.644214    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90ed78c_1eae_416d_8d2f_f4195c1c6b08.slice/crio-e2d1d52f2a2409d5d8121f776063190290d70207e81d96a84cd31866fc99f1a3 WatchSource:0}: Error finding container e2d1d52f2a2409d5d8121f776063190290d70207e81d96a84cd31866fc99f1a3: Status 404 returned error can't find the container with id e2d1d52f2a2409d5d8121f776063190290d70207e81d96a84cd31866fc99f1a3
Sep 17 21:59:46.736452 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:46.736422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:47.005446 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:47.005353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67dbddc5b4-5sp9t"]
Sep 17 21:59:47.008874 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:47.008847    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca563cf_0666_4ad1_8fce_b65ce76011de.slice/crio-18d3a857a5b583d3ee3af62c5f3b511f3c3406f4496c5ddd8390096865613b05 WatchSource:0}: Error finding container 18d3a857a5b583d3ee3af62c5f3b511f3c3406f4496c5ddd8390096865613b05: Status 404 returned error can't find the container with id 18d3a857a5b583d3ee3af62c5f3b511f3c3406f4496c5ddd8390096865613b05
Sep 17 21:59:47.047143 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:47.047084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dbddc5b4-5sp9t" event={"ID":"8ca563cf-0666-4ad1-8fce-b65ce76011de","Type":"ContainerStarted","Data":"18d3a857a5b583d3ee3af62c5f3b511f3c3406f4496c5ddd8390096865613b05"}
Sep 17 21:59:47.048745 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:47.048710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq" event={"ID":"eabdedc8-7b65-4b22-a838-42a79385bbec","Type":"ContainerStarted","Data":"8855c61b1cf93085143117df712b11565a97b8afd9f325b34dbd5224c4241b17"}
Sep 17 21:59:47.050077 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:47.050043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-7bvxc" event={"ID":"a90ed78c-1eae-416d-8d2f-f4195c1c6b08","Type":"ContainerStarted","Data":"e2d1d52f2a2409d5d8121f776063190290d70207e81d96a84cd31866fc99f1a3"}
Sep 17 21:59:52.948754 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:52.948724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:59:52.949182 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:52.948812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:59:52.951163 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:52.951137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a214a81a-0083-4cf0-8226-294fcc318e37-metrics-tls\") pod \"dns-default-x8hln\" (UID: \"a214a81a-0083-4cf0-8226-294fcc318e37\") " pod="openshift-dns/dns-default-x8hln"
Sep 17 21:59:52.951371 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:52.951354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"image-registry-7fdc6496f7-k2zhg\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") " pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:59:53.077885 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.077846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq" event={"ID":"eabdedc8-7b65-4b22-a838-42a79385bbec","Type":"ContainerStarted","Data":"569a8a007f3192988307848bb2a9a7aa208c6c6bc396edf99ca129d70c58a54f"}
Sep 17 21:59:53.079595 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.079562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dbddc5b4-5sp9t" event={"ID":"8ca563cf-0666-4ad1-8fce-b65ce76011de","Type":"ContainerStarted","Data":"8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba"}
Sep 17 21:59:53.092891 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.092851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-559958847d-qfwtq" podStartSLOduration=2.339693145 podStartE2EDuration="8.092834534s" totalImagesPullingTime="5.753141389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:53.092497088 +0000 UTC m=+99.575982823" watchObservedRunningTime="2026-09-17 21:59:53.092834534 +0000 UTC m=+99.576320266"
Sep 17 21:59:53.109213 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.109145    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67dbddc5b4-5sp9t" podStartSLOduration=1.811023185 podStartE2EDuration="7.109127484s" totalImagesPullingTime="5.298104299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:53.10897232 +0000 UTC m=+99.592458057" watchObservedRunningTime="2026-09-17 21:59:53.109127484 +0000 UTC m=+99.592613220"
Sep 17 21:59:53.211899 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.211871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hqrds\""
Sep 17 21:59:53.219946 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.219915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:59:53.224210 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.224189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb92b\""
Sep 17 21:59:53.233261 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.233231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x8hln"
Sep 17 21:59:53.503009 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.502917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fdc6496f7-k2zhg"]
Sep 17 21:59:53.507720 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:53.507691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee37e1e_49fd_42ae_bcbd_942a1a2898e7.slice/crio-c41e50d89d3adb6fa56635ebfb3dbc084c4d91122dde6ecc1f28cc9cf2c747bd WatchSource:0}: Error finding container c41e50d89d3adb6fa56635ebfb3dbc084c4d91122dde6ecc1f28cc9cf2c747bd: Status 404 returned error can't find the container with id c41e50d89d3adb6fa56635ebfb3dbc084c4d91122dde6ecc1f28cc9cf2c747bd
Sep 17 21:59:53.523372 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:53.523345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x8hln"]
Sep 17 21:59:53.526586 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:53.526544    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda214a81a_0083_4cf0_8226_294fcc318e37.slice/crio-dd1d9ebb561a5c0c91d02c117a3e1d81be0062956e6aa137c2221bb0aa95963d WatchSource:0}: Error finding container dd1d9ebb561a5c0c91d02c117a3e1d81be0062956e6aa137c2221bb0aa95963d: Status 404 returned error can't find the container with id dd1d9ebb561a5c0c91d02c117a3e1d81be0062956e6aa137c2221bb0aa95963d
Sep 17 21:59:54.085633 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:54.085586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x8hln" event={"ID":"a214a81a-0083-4cf0-8226-294fcc318e37","Type":"ContainerStarted","Data":"dd1d9ebb561a5c0c91d02c117a3e1d81be0062956e6aa137c2221bb0aa95963d"}
Sep 17 21:59:54.088134 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:54.088089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg" event={"ID":"bee37e1e-49fd-42ae-bcbd-942a1a2898e7","Type":"ContainerStarted","Data":"784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f"}
Sep 17 21:59:54.088254 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:54.088153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg" event={"ID":"bee37e1e-49fd-42ae-bcbd-942a1a2898e7","Type":"ContainerStarted","Data":"c41e50d89d3adb6fa56635ebfb3dbc084c4d91122dde6ecc1f28cc9cf2c747bd"}
Sep 17 21:59:54.088535 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:54.088518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 21:59:54.111734 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:54.111671    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg" podStartSLOduration=98.111656546 podStartE2EDuration="1m38.111656546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:54.109720668 +0000 UTC m=+100.593206404" watchObservedRunningTime="2026-09-17 21:59:54.111656546 +0000 UTC m=+100.595142279"
Sep 17 21:59:56.024904 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.024796    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d98c5dd95-mcgzd"]
Sep 17 21:59:56.026867 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.026846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.035092 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.035070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 21:59:56.038711 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.038690    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d98c5dd95-mcgzd"]
Sep 17 21:59:56.098022 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.097986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x8hln" event={"ID":"a214a81a-0083-4cf0-8226-294fcc318e37","Type":"ContainerStarted","Data":"3d623e2a6a0a9229c0d4fb8f8371a8aca4d8dcf50434801687292d15d60dc991"}
Sep 17 21:59:56.098168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.098044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x8hln" event={"ID":"a214a81a-0083-4cf0-8226-294fcc318e37","Type":"ContainerStarted","Data":"20078bb4efb31c11b7de89e1802436e5612baf47fa3beb1d25c958cdcbc443ed"}
Sep 17 21:59:56.117634 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.117593    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x8hln" podStartSLOduration=66.47112747 podStartE2EDuration="1m8.117578263s" totalImagesPullingTime="1.646450793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:56.116800838 +0000 UTC m=+102.600286573" watchObservedRunningTime="2026-09-17 21:59:56.117578263 +0000 UTC m=+102.601063997"
Sep 17 21:59:56.174122 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.174085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-oauth-serving-cert\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.174241 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.174216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-serving-cert\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.174367 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.174326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-trusted-ca-bundle\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.174499 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.174432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrchx\" (UniqueName: \"kubernetes.io/projected/104d8cc4-e4c3-40be-849e-98707b1c8dcd-kube-api-access-lrchx\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.174499 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.174475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-service-ca\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.174700 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.174526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-config\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.174700 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.174551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-oauth-config\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.275567 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.275499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-serving-cert\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.275567 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.275556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-trusted-ca-bundle\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.275802 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.275591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrchx\" (UniqueName: \"kubernetes.io/projected/104d8cc4-e4c3-40be-849e-98707b1c8dcd-kube-api-access-lrchx\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.275802 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.275617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-service-ca\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.275802 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.275644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-config\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.275802 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.275667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-oauth-config\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.275802 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.275704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-oauth-serving-cert\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.280126 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.276767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-service-ca\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.280126 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.276882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-config\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.280126 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.276900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-oauth-serving-cert\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.280126 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.276970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-trusted-ca-bundle\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.280126 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.279216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-oauth-config\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.280126 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.279421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-serving-cert\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.285304 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.285261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrchx\" (UniqueName: \"kubernetes.io/projected/104d8cc4-e4c3-40be-849e-98707b1c8dcd-kube-api-access-lrchx\") pod \"console-6d98c5dd95-mcgzd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") " pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.337647 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.337619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 21:59:56.616869 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.616795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d98c5dd95-mcgzd"]
Sep 17 21:59:56.620340 ip-10-0-139-88 kubenswrapper[2563]: W0917 21:59:56.620310    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104d8cc4_e4c3_40be_849e_98707b1c8dcd.slice/crio-11e732f1b97ee155a794c79c6c1675e7d90fe13a77d32d8bb5d1bc57cee31d68 WatchSource:0}: Error finding container 11e732f1b97ee155a794c79c6c1675e7d90fe13a77d32d8bb5d1bc57cee31d68: Status 404 returned error can't find the container with id 11e732f1b97ee155a794c79c6c1675e7d90fe13a77d32d8bb5d1bc57cee31d68
Sep 17 21:59:56.737298 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.737264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:56.737458 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.737359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 21:59:56.739383 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.739140    2563 patch_prober.go:28] interesting pod/console-67dbddc5b4-5sp9t container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body=
Sep 17 21:59:56.739383 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:56.739196    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-67dbddc5b4-5sp9t" podUID="8ca563cf-0666-4ad1-8fce-b65ce76011de" containerName="console" probeResult="failure" output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused"
Sep 17 21:59:57.102781 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:57.102742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d98c5dd95-mcgzd" event={"ID":"104d8cc4-e4c3-40be-849e-98707b1c8dcd","Type":"ContainerStarted","Data":"e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2"}
Sep 17 21:59:57.103237 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:57.102799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d98c5dd95-mcgzd" event={"ID":"104d8cc4-e4c3-40be-849e-98707b1c8dcd","Type":"ContainerStarted","Data":"11e732f1b97ee155a794c79c6c1675e7d90fe13a77d32d8bb5d1bc57cee31d68"}
Sep 17 21:59:57.103237 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:57.103132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x8hln"
Sep 17 21:59:57.122726 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:57.122682    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d98c5dd95-mcgzd" podStartSLOduration=1.122669011 podStartE2EDuration="1.122669011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:57.121738353 +0000 UTC m=+103.605224079" watchObservedRunningTime="2026-09-17 21:59:57.122669011 +0000 UTC m=+103.606154742"
Sep 17 21:59:57.984927 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:57.984898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ssgcr"
Sep 17 21:59:59.439511 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.439480    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rxq5z"]
Sep 17 21:59:59.473489 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.473463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.476485 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.476456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 21:59:59.476651 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.476489    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 21:59:59.476893 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.476873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krpp4\""
Sep 17 21:59:59.476992 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.476913    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 21:59:59.477745 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.477546    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 21:59:59.477745 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.477593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 21:59:59.477941 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.477835    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 21:59:59.603969 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.603934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604154 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.603979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef457f63-4426-4082-9ee8-398a6210329e-metrics-client-ca\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604154 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.603998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fsfg\" (UniqueName: \"kubernetes.io/projected/ef457f63-4426-4082-9ee8-398a6210329e-kube-api-access-6fsfg\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604154 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.604060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604154 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.604114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-wtmp\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604366 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.604182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-tls\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604366 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.604226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-sys\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604366 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.604253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-root\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.604366 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.604312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-textfile\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.704868 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.704785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef457f63-4426-4082-9ee8-398a6210329e-metrics-client-ca\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.704868 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.704828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fsfg\" (UniqueName: \"kubernetes.io/projected/ef457f63-4426-4082-9ee8-398a6210329e-kube-api-access-6fsfg\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705096 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.704963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705096 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-wtmp\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705096 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-tls\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705096 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-sys\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705341 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-root\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705341 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-textfile\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705341 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705611 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-wtmp\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705611 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef457f63-4426-4082-9ee8-398a6210329e-metrics-client-ca\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705611 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-root\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705611 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef457f63-4426-4082-9ee8-398a6210329e-sys\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705861 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.705861 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.705726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-textfile\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.707863 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.707839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.707981 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.707931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef457f63-4426-4082-9ee8-398a6210329e-node-exporter-tls\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.712505 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.712481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fsfg\" (UniqueName: \"kubernetes.io/projected/ef457f63-4426-4082-9ee8-398a6210329e-kube-api-access-6fsfg\") pod \"node-exporter-rxq5z\" (UID: \"ef457f63-4426-4082-9ee8-398a6210329e\") " pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 21:59:59.784168 ip-10-0-139-88 kubenswrapper[2563]: I0917 21:59:59.784140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rxq5z"
Sep 17 22:00:01.461392 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.461354    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"]
Sep 17 22:00:01.479818 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.479786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"]
Sep 17 22:00:01.480002 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.479902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.482647 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.482613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 22:00:01.482782 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.482715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nfbcx\""
Sep 17 22:00:01.482782 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.482761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 22:00:01.482782 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.482767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 22:00:01.483004 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.482950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 22:00:01.483076 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.482950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 22:00:01.483076 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.482835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-65ua8ic8a79jg\""
Sep 17 22:00:01.621394 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.621572 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-tls\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.621572 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2sjj\" (UniqueName: \"kubernetes.io/projected/15c4cfd7-f383-46e9-95df-4e1345a67125-kube-api-access-g2sjj\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.621695 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.621695 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c4cfd7-f383-46e9-95df-4e1345a67125-metrics-client-ca\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.621695 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-grpc-tls\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.621695 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.621865 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.621727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722540 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722540 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722751 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-tls\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722751 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2sjj\" (UniqueName: \"kubernetes.io/projected/15c4cfd7-f383-46e9-95df-4e1345a67125-kube-api-access-g2sjj\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722751 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722751 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c4cfd7-f383-46e9-95df-4e1345a67125-metrics-client-ca\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722751 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-grpc-tls\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.722751 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.722733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.723578 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.723453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c4cfd7-f383-46e9-95df-4e1345a67125-metrics-client-ca\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.725633 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.725583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.725633 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.725620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.725905 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.725882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.725987 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.725957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.726335 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.726313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-grpc-tls\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.726879 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.726856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15c4cfd7-f383-46e9-95df-4e1345a67125-secret-thanos-querier-tls\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.730598 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.730577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2sjj\" (UniqueName: \"kubernetes.io/projected/15c4cfd7-f383-46e9-95df-4e1345a67125-kube-api-access-g2sjj\") pod \"thanos-querier-7b4cbdc4cd-jmdbm\" (UID: \"15c4cfd7-f383-46e9-95df-4e1345a67125\") " pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:01.790763 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:01.790717    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:03.711847 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:00:03.711819    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef457f63_4426_4082_9ee8_398a6210329e.slice/crio-cc8a8ee518e5311d02dc300efacc6b130204892925124639f2e07ec8b05d82f1 WatchSource:0}: Error finding container cc8a8ee518e5311d02dc300efacc6b130204892925124639f2e07ec8b05d82f1: Status 404 returned error can't find the container with id cc8a8ee518e5311d02dc300efacc6b130204892925124639f2e07ec8b05d82f1
Sep 17 22:00:03.982409 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:03.982377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"]
Sep 17 22:00:04.131907 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.131865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-7bvxc" event={"ID":"a90ed78c-1eae-416d-8d2f-f4195c1c6b08","Type":"ContainerStarted","Data":"7bf37928471add084e6769bd7ae8ec7f01bc7b3361bf0346bc63c5f1306187d7"}
Sep 17 22:00:04.132091 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.132058    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-7bvxc"
Sep 17 22:00:04.133603 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.133567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxq5z" event={"ID":"ef457f63-4426-4082-9ee8-398a6210329e","Type":"ContainerStarted","Data":"cc8a8ee518e5311d02dc300efacc6b130204892925124639f2e07ec8b05d82f1"}
Sep 17 22:00:04.135070 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.135046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" event={"ID":"15c4cfd7-f383-46e9-95df-4e1345a67125","Type":"ContainerStarted","Data":"7a2855fa7c3b955899e831118071ebeae6ebad4d4a13b6daaaed33d8201206db"}
Sep 17 22:00:04.152607 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.152538    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-7bvxc" podStartSLOduration=1.011736477 podStartE2EDuration="18.15251756s" totalImagesPullingTime="17.140781083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:04.151363245 +0000 UTC m=+110.634848980" watchObservedRunningTime="2026-09-17 22:00:04.15251756 +0000 UTC m=+110.636003296"
Sep 17 22:00:04.154063 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.154041    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-7bvxc"
Sep 17 22:00:04.641451 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.641409    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5884778dd9-rdgf9"]
Sep 17 22:00:04.644675 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.644649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.647436 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.647408    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zm7wz\""
Sep 17 22:00:04.647582 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.647443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 22:00:04.647582 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.647509    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 22:00:04.647582 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.647569    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 22:00:04.647729 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.647569    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 22:00:04.648677 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.648655    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 22:00:04.654952 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.654926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 22:00:04.659263 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.659241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5884778dd9-rdgf9"]
Sep 17 22:00:04.748029 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.747778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.748029 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.747833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-secret-telemeter-client\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.748029 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.747876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.748029 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.747934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-serving-certs-ca-bundle\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.748029 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.747967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-federate-client-tls\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.748978 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.748007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvhd\" (UniqueName: \"kubernetes.io/projected/210f2bf8-3895-4462-9f7b-8811a3785887-kube-api-access-pcvhd\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.748978 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.748889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-telemeter-client-tls\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.749241 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.748935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-metrics-client-ca\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.850561 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-serving-certs-ca-bundle\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.850748 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-federate-client-tls\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.850748 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvhd\" (UniqueName: \"kubernetes.io/projected/210f2bf8-3895-4462-9f7b-8811a3785887-kube-api-access-pcvhd\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.850748 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-telemeter-client-tls\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.850748 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-metrics-client-ca\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.852173 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.852173 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-secret-telemeter-client\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.852173 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.850863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.852173 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.851983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-serving-certs-ca-bundle\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.852513 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.852488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-metrics-client-ca\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.852683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.852636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210f2bf8-3895-4462-9f7b-8811a3785887-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.855034 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.854995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.855390 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.855349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-telemeter-client-tls\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.855964 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.855830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-secret-telemeter-client\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.855964 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.855926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/210f2bf8-3895-4462-9f7b-8811a3785887-federate-client-tls\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.861096 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.860918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvhd\" (UniqueName: \"kubernetes.io/projected/210f2bf8-3895-4462-9f7b-8811a3785887-kube-api-access-pcvhd\") pod \"telemeter-client-5884778dd9-rdgf9\" (UID: \"210f2bf8-3895-4462-9f7b-8811a3785887\") " pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:04.957966 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:04.957929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9"
Sep 17 22:00:05.557032 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.556949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5884778dd9-rdgf9"]
Sep 17 22:00:05.643436 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:00:05.643395    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210f2bf8_3895_4462_9f7b_8811a3785887.slice/crio-4aa49e7c223f34786dde3f39eea267c91e4fc37d4cd49323eb767f2fe873557c WatchSource:0}: Error finding container 4aa49e7c223f34786dde3f39eea267c91e4fc37d4cd49323eb767f2fe873557c: Status 404 returned error can't find the container with id 4aa49e7c223f34786dde3f39eea267c91e4fc37d4cd49323eb767f2fe873557c
Sep 17 22:00:05.721798 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.721768    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:05.746397 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.746363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.755927 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.755898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 22:00:05.759559 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.755912    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 22:00:05.762388 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.757697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 22:00:05.762784 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.758267    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 22:00:05.763043 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.758970    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 22:00:05.764133 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.763383    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 22:00:05.764133 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.759871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 22:00:05.764133 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.760889    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wnzmx\""
Sep 17 22:00:05.764133 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.760220    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1srvlm1hkh49u\""
Sep 17 22:00:05.764133 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.761257    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 22:00:05.764498 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.760987    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 22:00:05.766082 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.761548    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 22:00:05.766082 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.761366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 22:00:05.766082 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.761628    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 22:00:05.766082 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.766079    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 22:00:05.767987 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.767841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:05.773787 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.773763    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 22:00:05.774718 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.774697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 22:00:05.862791 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.862698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.862791 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.862769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.862985 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.862798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.862985 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.862841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.862985 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.862882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.862985 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.862915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.862985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863419 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ed5a58f-f659-493f-a5fc-ecfd1b514221-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863419 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863419 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863419 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ed5a58f-f659-493f-a5fc-ecfd1b514221-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863610 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-config\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863610 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863610 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863610 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.863870 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.863619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-922st\" (UniqueName: \"kubernetes.io/projected/5ed5a58f-f659-493f-a5fc-ecfd1b514221-kube-api-access-922st\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.903868 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:00:05.903827    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef457f63_4426_4082_9ee8_398a6210329e.slice/crio-conmon-189d869cdd32afa17b69fcfb5d9a644176e0316ee75cce3ec2302acbaae4064f.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 22:00:05.964817 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.964783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.964817 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.964826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-922st\" (UniqueName: \"kubernetes.io/projected/5ed5a58f-f659-493f-a5fc-ecfd1b514221-kube-api-access-922st\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965045 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.964967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965045 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965182 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965182 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965182 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965377 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965377 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965377 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965377 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965377 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ed5a58f-f659-493f-a5fc-ecfd1b514221-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ed5a58f-f659-493f-a5fc-ecfd1b514221-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-config\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.965617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.965574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.966061 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.966016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.966147 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.966098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.966935 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.966904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.968468 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.967815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.969501 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.968899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.969501 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.969125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.969501 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.969237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.969501 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.969392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.969501 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.969450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.969800 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.969739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.972790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.972334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.973022 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.973000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.973345 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.973320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.973423 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.973320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ed5a58f-f659-493f-a5fc-ecfd1b514221-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.975687 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.975338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-config\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.975687 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.975338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ed5a58f-f659-493f-a5fc-ecfd1b514221-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.975687 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.975649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-922st\" (UniqueName: \"kubernetes.io/projected/5ed5a58f-f659-493f-a5fc-ecfd1b514221-kube-api-access-922st\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.976573 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.976118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ed5a58f-f659-493f-a5fc-ecfd1b514221-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.976678 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.976570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:05.977165 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:05.977093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ed5a58f-f659-493f-a5fc-ecfd1b514221-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ed5a58f-f659-493f-a5fc-ecfd1b514221\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:06.076084 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.075855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:06.146355 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.146207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9" event={"ID":"210f2bf8-3895-4462-9f7b-8811a3785887","Type":"ContainerStarted","Data":"4aa49e7c223f34786dde3f39eea267c91e4fc37d4cd49323eb767f2fe873557c"}
Sep 17 22:00:06.151524 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.150774    2563 generic.go:356] "Generic (PLEG): container finished" podID="ef457f63-4426-4082-9ee8-398a6210329e" containerID="189d869cdd32afa17b69fcfb5d9a644176e0316ee75cce3ec2302acbaae4064f" exitCode=0
Sep 17 22:00:06.151524 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.150902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxq5z" event={"ID":"ef457f63-4426-4082-9ee8-398a6210329e","Type":"ContainerDied","Data":"189d869cdd32afa17b69fcfb5d9a644176e0316ee75cce3ec2302acbaae4064f"}
Sep 17 22:00:06.338718 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.338692    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 22:00:06.339038 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.338986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 22:00:06.340604 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.340497    2563 patch_prober.go:28] interesting pod/console-6d98c5dd95-mcgzd container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body=
Sep 17 22:00:06.340604 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.340545    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6d98c5dd95-mcgzd" podUID="104d8cc4-e4c3-40be-849e-98707b1c8dcd" containerName="console" probeResult="failure" output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused"
Sep 17 22:00:06.401088 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.401058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:06.406979 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:00:06.406948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed5a58f_f659_493f_a5fc_ecfd1b514221.slice/crio-4de4695bffd4df08d1e28b87b50b7d45850cb423e65e2d3802eca175f377d0d2 WatchSource:0}: Error finding container 4de4695bffd4df08d1e28b87b50b7d45850cb423e65e2d3802eca175f377d0d2: Status 404 returned error can't find the container with id 4de4695bffd4df08d1e28b87b50b7d45850cb423e65e2d3802eca175f377d0d2
Sep 17 22:00:06.441544 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:06.441371    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67dbddc5b4-5sp9t"]
Sep 17 22:00:07.110714 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:07.110590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x8hln"
Sep 17 22:00:07.181475 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:07.180316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxq5z" event={"ID":"ef457f63-4426-4082-9ee8-398a6210329e","Type":"ContainerStarted","Data":"8a5965d690fe1b5c53614eefa62a4157a062bd8e09ed6f6c75166a0c96f40a1d"}
Sep 17 22:00:07.181475 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:07.180429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxq5z" event={"ID":"ef457f63-4426-4082-9ee8-398a6210329e","Type":"ContainerStarted","Data":"372ae87face0d803b35d399fb827bba6bdc38c8b73241e558ba3f05a45a7b4d3"}
Sep 17 22:00:07.184729 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:07.184648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerStarted","Data":"4de4695bffd4df08d1e28b87b50b7d45850cb423e65e2d3802eca175f377d0d2"}
Sep 17 22:00:07.202139 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:07.202074    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rxq5z" podStartSLOduration=6.190880616 podStartE2EDuration="8.202056563s" totalImagesPullingTime="1.934111159s" totalInitContainerRuntime="77.064788ms" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:07.201662446 +0000 UTC m=+113.685148192" watchObservedRunningTime="2026-09-17 22:00:07.202056563 +0000 UTC m=+113.685542299"
Sep 17 22:00:07.892146 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:07.892097    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fdc6496f7-k2zhg"]
Sep 17 22:00:09.197537 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:09.197450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" event={"ID":"15c4cfd7-f383-46e9-95df-4e1345a67125","Type":"ContainerStarted","Data":"7d3db1dd7e94fc91a8159619e570c05054b75f02a6acd04fbbd3bba02dab79d8"}
Sep 17 22:00:10.203910 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:10.203823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" event={"ID":"15c4cfd7-f383-46e9-95df-4e1345a67125","Type":"ContainerStarted","Data":"72352a9d02e05a4894af6dda88b0292d52f829b833808bda342c14b54b97d3d3"}
Sep 17 22:00:10.203910 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:10.203906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" event={"ID":"15c4cfd7-f383-46e9-95df-4e1345a67125","Type":"ContainerStarted","Data":"350376af40806414192ca5bd1c72a0734ebe4eb08004cf52ffabaa29269fde43"}
Sep 17 22:00:10.205660 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:10.205627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9" event={"ID":"210f2bf8-3895-4462-9f7b-8811a3785887","Type":"ContainerStarted","Data":"bad98545691eed5870be615d088ea63f86ae89aedfb4fc7bb10707e900ba5a59"}
Sep 17 22:00:10.207443 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:10.207411    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ed5a58f-f659-493f-a5fc-ecfd1b514221" containerID="d4295a2cd766ceb02f8efb9cb259593956af6f617da5d58c28a0c3271176acc2" exitCode=0
Sep 17 22:00:10.207550 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:10.207465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerDied","Data":"d4295a2cd766ceb02f8efb9cb259593956af6f617da5d58c28a0c3271176acc2"}
Sep 17 22:00:11.215117 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:11.215071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9" event={"ID":"210f2bf8-3895-4462-9f7b-8811a3785887","Type":"ContainerStarted","Data":"8294adcefa99a22e6a13a70185ab2351246b0d4baeacef417742cc9180f00324"}
Sep 17 22:00:11.215578 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:11.215158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9" event={"ID":"210f2bf8-3895-4462-9f7b-8811a3785887","Type":"ContainerStarted","Data":"1be4488ec40fc432df36cbe09e13a2275dbbdc814979648f0c2693c3d563bf34"}
Sep 17 22:00:11.236482 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:11.236423    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5884778dd9-rdgf9" podStartSLOduration=2.66275439 podStartE2EDuration="7.236399084s" totalImagesPullingTime="4.573644694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:11.234432766 +0000 UTC m=+117.717918501" watchObservedRunningTime="2026-09-17 22:00:11.236399084 +0000 UTC m=+117.719884818"
Sep 17 22:00:11.431207 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:11.431176    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d98c5dd95-mcgzd"]
Sep 17 22:00:12.229464 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:12.229422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" event={"ID":"15c4cfd7-f383-46e9-95df-4e1345a67125","Type":"ContainerStarted","Data":"d527b234f1a528228c7c252935c4e114b79e9074e6a9baedbb074655c78ad390"}
Sep 17 22:00:12.230076 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:12.230051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:12.230207 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:12.230143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" event={"ID":"15c4cfd7-f383-46e9-95df-4e1345a67125","Type":"ContainerStarted","Data":"faf7389e3b8fc9e587f377bd2278a8cdf16ad9d6bc298612ab4af779821b6237"}
Sep 17 22:00:12.230207 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:12.230177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" event={"ID":"15c4cfd7-f383-46e9-95df-4e1345a67125","Type":"ContainerStarted","Data":"89a79e740f415e83e0ca29ff75a73ed074c2ccb188b9ccac8cb142722b9e2e47"}
Sep 17 22:00:12.254072 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:12.254022    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm" podStartSLOduration=4.801197461 podStartE2EDuration="11.254000883s" totalImagesPullingTime="6.452803422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:12.252068948 +0000 UTC m=+118.735554683" watchObservedRunningTime="2026-09-17 22:00:12.254000883 +0000 UTC m=+118.737486618"
Sep 17 22:00:14.242838 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:14.242812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b4cbdc4cd-jmdbm"
Sep 17 22:00:16.248780 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:16.248738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerStarted","Data":"aae97c06b240086dd249f31a285ea9536ffcb7dba64b71361697e1fc61c226e6"}
Sep 17 22:00:17.257581 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:17.257548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerStarted","Data":"cd3b7de0f1775e443dca7b917a9411a4641c2a037371518550297a2e95428fe6"}
Sep 17 22:00:17.257958 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:17.257609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerStarted","Data":"194fbae784b1cf6f13429fe60867866e4582d0b8ac7c6a1fa29c10d36b8560f7"}
Sep 17 22:00:17.257958 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:17.257635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerStarted","Data":"4cafd674474b7c1120bb4dfdfd1e9448b9c2de54a55ef351f31ace19c36df7dc"}
Sep 17 22:00:17.257958 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:17.257658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerStarted","Data":"585afe0d1425e6dd6b223bb50f942aef5370fdb1a7f80fa630c1f0b1a103b93d"}
Sep 17 22:00:17.257958 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:17.257680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ed5a58f-f659-493f-a5fc-ecfd1b514221","Type":"ContainerStarted","Data":"b83dd85befd32c17016a7ee24d170c78ffa3a5812dc58aba4a8c40dfe641827a"}
Sep 17 22:00:17.288850 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:17.288795    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.088424391 podStartE2EDuration="12.288780326s" totalImagesPullingTime="9.180135477s" totalInitContainerRuntime="20.220458ms" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:17.286447565 +0000 UTC m=+123.769933301" watchObservedRunningTime="2026-09-17 22:00:17.288780326 +0000 UTC m=+123.772266059"
Sep 17 22:00:17.898153 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:17.898119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 22:00:21.076222 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:21.076187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:25.557778 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:25.557745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 22:00:25.559959 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:25.559940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cceb95-3224-4dd3-a4e2-d50bf1c58950-metrics-certs\") pod \"network-metrics-daemon-r22wg\" (UID: \"92cceb95-3224-4dd3-a4e2-d50bf1c58950\") " pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 22:00:25.833641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:25.833557    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzq2x\""
Sep 17 22:00:25.841836 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:25.841815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r22wg"
Sep 17 22:00:26.130801 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:26.130717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r22wg"]
Sep 17 22:00:26.133443 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:00:26.133417    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cceb95_3224_4dd3_a4e2_d50bf1c58950.slice/crio-f4eb797385b0a12e8c37ae878d0d48d8988a73c4aa7b16fe7b8b7d6161e50bb2 WatchSource:0}: Error finding container f4eb797385b0a12e8c37ae878d0d48d8988a73c4aa7b16fe7b8b7d6161e50bb2: Status 404 returned error can't find the container with id f4eb797385b0a12e8c37ae878d0d48d8988a73c4aa7b16fe7b8b7d6161e50bb2
Sep 17 22:00:26.296448 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:26.296414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r22wg" event={"ID":"92cceb95-3224-4dd3-a4e2-d50bf1c58950","Type":"ContainerStarted","Data":"f4eb797385b0a12e8c37ae878d0d48d8988a73c4aa7b16fe7b8b7d6161e50bb2"}
Sep 17 22:00:28.306899 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:28.306865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r22wg" event={"ID":"92cceb95-3224-4dd3-a4e2-d50bf1c58950","Type":"ContainerStarted","Data":"b527b33002908f80664f5bcec3561ebee5411cc510fb617f175d502a0881f054"}
Sep 17 22:00:28.307463 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:28.306922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r22wg" event={"ID":"92cceb95-3224-4dd3-a4e2-d50bf1c58950","Type":"ContainerStarted","Data":"0efa172576717fefe376e2e394f7eaf4659b4774c23976bcf6c87ba267d4786b"}
Sep 17 22:00:28.329570 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:28.329528    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r22wg" podStartSLOduration=130.724680141 podStartE2EDuration="2m12.329512999s" totalImagesPullingTime="1.604832858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:28.32834066 +0000 UTC m=+134.811826393" watchObservedRunningTime="2026-09-17 22:00:28.329512999 +0000 UTC m=+134.812998732"
Sep 17 22:00:31.470831 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.470755    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67dbddc5b4-5sp9t" podUID="8ca563cf-0666-4ad1-8fce-b65ce76011de" containerName="console" containerID="cri-o://8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba" gracePeriod=15
Sep 17 22:00:31.740277 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.740219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 22:00:31.808328 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t78s\" (UniqueName: \"kubernetes.io/projected/8ca563cf-0666-4ad1-8fce-b65ce76011de-kube-api-access-2t78s\") pod \"8ca563cf-0666-4ad1-8fce-b65ce76011de\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") "
Sep 17 22:00:31.808464 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-serving-cert\") pod \"8ca563cf-0666-4ad1-8fce-b65ce76011de\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") "
Sep 17 22:00:31.808464 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-service-ca\") pod \"8ca563cf-0666-4ad1-8fce-b65ce76011de\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") "
Sep 17 22:00:31.808464 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-oauth-serving-cert\") pod \"8ca563cf-0666-4ad1-8fce-b65ce76011de\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") "
Sep 17 22:00:31.808621 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-config\") pod \"8ca563cf-0666-4ad1-8fce-b65ce76011de\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") "
Sep 17 22:00:31.808621 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-oauth-config\") pod \"8ca563cf-0666-4ad1-8fce-b65ce76011de\" (UID: \"8ca563cf-0666-4ad1-8fce-b65ce76011de\") "
Sep 17 22:00:31.808876 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808836    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8ca563cf-0666-4ad1-8fce-b65ce76011de" (UID: "8ca563cf-0666-4ad1-8fce-b65ce76011de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:31.808876 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808856    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-service-ca" (OuterVolumeSpecName: "service-ca") pod "8ca563cf-0666-4ad1-8fce-b65ce76011de" (UID: "8ca563cf-0666-4ad1-8fce-b65ce76011de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:31.809030 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.808946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-config" (OuterVolumeSpecName: "console-config") pod "8ca563cf-0666-4ad1-8fce-b65ce76011de" (UID: "8ca563cf-0666-4ad1-8fce-b65ce76011de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:31.810584 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.810556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8ca563cf-0666-4ad1-8fce-b65ce76011de" (UID: "8ca563cf-0666-4ad1-8fce-b65ce76011de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:31.810675 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.810579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca563cf-0666-4ad1-8fce-b65ce76011de-kube-api-access-2t78s" (OuterVolumeSpecName: "kube-api-access-2t78s") pod "8ca563cf-0666-4ad1-8fce-b65ce76011de" (UID: "8ca563cf-0666-4ad1-8fce-b65ce76011de"). InnerVolumeSpecName "kube-api-access-2t78s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:31.810675 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.810590    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8ca563cf-0666-4ad1-8fce-b65ce76011de" (UID: "8ca563cf-0666-4ad1-8fce-b65ce76011de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:31.909384 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.909358    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:31.909384 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.909380    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-oauth-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:31.909531 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.909410    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:31.909531 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.909420    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-oauth-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:31.909531 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.909430    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t78s\" (UniqueName: \"kubernetes.io/projected/8ca563cf-0666-4ad1-8fce-b65ce76011de-kube-api-access-2t78s\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:31.909531 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:31.909438    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca563cf-0666-4ad1-8fce-b65ce76011de-console-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:32.322454 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.322424    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ca563cf-0666-4ad1-8fce-b65ce76011de" containerID="8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba" exitCode=0
Sep 17 22:00:32.322627 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.322465    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dbddc5b4-5sp9t"
Sep 17 22:00:32.322627 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.322506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dbddc5b4-5sp9t" event={"ID":"8ca563cf-0666-4ad1-8fce-b65ce76011de","Type":"ContainerDied","Data":"8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba"}
Sep 17 22:00:32.322627 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.322557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dbddc5b4-5sp9t" event={"ID":"8ca563cf-0666-4ad1-8fce-b65ce76011de","Type":"ContainerDied","Data":"18d3a857a5b583d3ee3af62c5f3b511f3c3406f4496c5ddd8390096865613b05"}
Sep 17 22:00:32.322627 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.322588    2563 scope.go:117] "RemoveContainer" containerID="8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba"
Sep 17 22:00:32.333827 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.333811    2563 scope.go:117] "RemoveContainer" containerID="8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba"
Sep 17 22:00:32.334069 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:00:32.334052    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba\": container with ID starting with 8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba not found: ID does not exist" containerID="8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba"
Sep 17 22:00:32.334147 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.334078    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba"} err="failed to get container status \"8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba\": rpc error: code = NotFound desc = could not find container \"8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba\": container with ID starting with 8bff57b734cc36ddae2b15ce296b6e5e1dc978a85a2fb565ef96d63ae81f3cba not found: ID does not exist"
Sep 17 22:00:32.344864 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.344840    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67dbddc5b4-5sp9t"]
Sep 17 22:00:32.347315 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.347296    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67dbddc5b4-5sp9t"]
Sep 17 22:00:32.913898 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:32.913862    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg" podUID="bee37e1e-49fd-42ae-bcbd-942a1a2898e7" containerName="registry" containerID="cri-o://784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f" gracePeriod=30
Sep 17 22:00:33.152101 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.152079    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 22:00:33.222030 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-installation-pull-secrets\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgmks\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-kube-api-access-kgmks\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-image-registry-private-configuration\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222353 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-certificates\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222353 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222303    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-trusted-ca\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222353 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-bound-sa-token\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222499 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222372    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-ca-trust-extracted\") pod \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\" (UID: \"bee37e1e-49fd-42ae-bcbd-942a1a2898e7\") "
Sep 17 22:00:33.222680 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222653    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:33.222680 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.222667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:33.224545 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.224497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:33.224674 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.224579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:33.224745 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.224731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-kube-api-access-kgmks" (OuterVolumeSpecName: "kube-api-access-kgmks") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "kube-api-access-kgmks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:33.224745 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.224730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:33.224937 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.224923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:33.233382 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.233358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bee37e1e-49fd-42ae-bcbd-942a1a2898e7" (UID: "bee37e1e-49fd-42ae-bcbd-942a1a2898e7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:00:33.323553 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323529    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-trusted-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.323553 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323552    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-bound-sa-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.323669 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323566    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-ca-trust-extracted\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.323669 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323574    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.323669 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323583    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-installation-pull-secrets\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.323669 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323591    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgmks\" (UniqueName: \"kubernetes.io/projected/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-kube-api-access-kgmks\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.323669 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323603    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-image-registry-private-configuration\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.323669 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.323612    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bee37e1e-49fd-42ae-bcbd-942a1a2898e7-registry-certificates\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:33.328570 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.328542    2563 generic.go:356] "Generic (PLEG): container finished" podID="bee37e1e-49fd-42ae-bcbd-942a1a2898e7" containerID="784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f" exitCode=0
Sep 17 22:00:33.328657 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.328585    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg"
Sep 17 22:00:33.328657 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.328597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg" event={"ID":"bee37e1e-49fd-42ae-bcbd-942a1a2898e7","Type":"ContainerDied","Data":"784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f"}
Sep 17 22:00:33.328657 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.328650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fdc6496f7-k2zhg" event={"ID":"bee37e1e-49fd-42ae-bcbd-942a1a2898e7","Type":"ContainerDied","Data":"c41e50d89d3adb6fa56635ebfb3dbc084c4d91122dde6ecc1f28cc9cf2c747bd"}
Sep 17 22:00:33.328772 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.328678    2563 scope.go:117] "RemoveContainer" containerID="784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f"
Sep 17 22:00:33.337328 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.337301    2563 scope.go:117] "RemoveContainer" containerID="784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f"
Sep 17 22:00:33.337573 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:00:33.337553    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f\": container with ID starting with 784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f not found: ID does not exist" containerID="784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f"
Sep 17 22:00:33.337617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.337582    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f"} err="failed to get container status \"784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f\": rpc error: code = NotFound desc = could not find container \"784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f\": container with ID starting with 784789bcef302a81241b40e8fb73a921c639bc6266a759b76339b86aa800ff1f not found: ID does not exist"
Sep 17 22:00:33.350906 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.350881    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fdc6496f7-k2zhg"]
Sep 17 22:00:33.355010 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.354989    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fdc6496f7-k2zhg"]
Sep 17 22:00:33.710575 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.710546    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca563cf-0666-4ad1-8fce-b65ce76011de" path="/var/lib/kubelet/pods/8ca563cf-0666-4ad1-8fce-b65ce76011de/volumes"
Sep 17 22:00:33.710932 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:33.710919    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee37e1e-49fd-42ae-bcbd-942a1a2898e7" path="/var/lib/kubelet/pods/bee37e1e-49fd-42ae-bcbd-942a1a2898e7/volumes"
Sep 17 22:00:35.338570 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:35.338529    2563 generic.go:356] "Generic (PLEG): container finished" podID="264fbc4a-e424-4eff-9d4c-0ba1e486a36e" containerID="17806305da0f27dfefb6f6e81bbddcc0206ce620b26400d03d1c97f81597249b" exitCode=0
Sep 17 22:00:35.338930 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:35.338598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz" event={"ID":"264fbc4a-e424-4eff-9d4c-0ba1e486a36e","Type":"ContainerDied","Data":"17806305da0f27dfefb6f6e81bbddcc0206ce620b26400d03d1c97f81597249b"}
Sep 17 22:00:35.338930 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:35.338925    2563 scope.go:117] "RemoveContainer" containerID="17806305da0f27dfefb6f6e81bbddcc0206ce620b26400d03d1c97f81597249b"
Sep 17 22:00:36.344376 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.344345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4ffmz" event={"ID":"264fbc4a-e424-4eff-9d4c-0ba1e486a36e","Type":"ContainerStarted","Data":"cd9237a209f3dbd32a5112858f8be626d8e7069cbf24e950d2356b2445305151"}
Sep 17 22:00:36.468560 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.468525    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d98c5dd95-mcgzd" podUID="104d8cc4-e4c3-40be-849e-98707b1c8dcd" containerName="console" containerID="cri-o://e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2" gracePeriod=15
Sep 17 22:00:36.708072 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.708049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 22:00:36.851357 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-trusted-ca-bundle\") pod \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") "
Sep 17 22:00:36.851523 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851407    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-serving-cert\") pod \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") "
Sep 17 22:00:36.851605 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-oauth-serving-cert\") pod \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") "
Sep 17 22:00:36.851658 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-oauth-config\") pod \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") "
Sep 17 22:00:36.851658 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-service-ca\") pod \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") "
Sep 17 22:00:36.851758 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-config\") pod \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") "
Sep 17 22:00:36.851758 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "104d8cc4-e4c3-40be-849e-98707b1c8dcd" (UID: "104d8cc4-e4c3-40be-849e-98707b1c8dcd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:36.851758 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrchx\" (UniqueName: \"kubernetes.io/projected/104d8cc4-e4c3-40be-849e-98707b1c8dcd-kube-api-access-lrchx\") pod \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\" (UID: \"104d8cc4-e4c3-40be-849e-98707b1c8dcd\") "
Sep 17 22:00:36.852014 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "104d8cc4-e4c3-40be-849e-98707b1c8dcd" (UID: "104d8cc4-e4c3-40be-849e-98707b1c8dcd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:36.852014 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.851991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-service-ca" (OuterVolumeSpecName: "service-ca") pod "104d8cc4-e4c3-40be-849e-98707b1c8dcd" (UID: "104d8cc4-e4c3-40be-849e-98707b1c8dcd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:36.852247 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.852059    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-oauth-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:36.852247 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.852077    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-trusted-ca-bundle\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:36.852247 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.852137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-config" (OuterVolumeSpecName: "console-config") pod "104d8cc4-e4c3-40be-849e-98707b1c8dcd" (UID: "104d8cc4-e4c3-40be-849e-98707b1c8dcd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:36.853828 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.853802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104d8cc4-e4c3-40be-849e-98707b1c8dcd-kube-api-access-lrchx" (OuterVolumeSpecName: "kube-api-access-lrchx") pod "104d8cc4-e4c3-40be-849e-98707b1c8dcd" (UID: "104d8cc4-e4c3-40be-849e-98707b1c8dcd"). InnerVolumeSpecName "kube-api-access-lrchx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:36.853964 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.853946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "104d8cc4-e4c3-40be-849e-98707b1c8dcd" (UID: "104d8cc4-e4c3-40be-849e-98707b1c8dcd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:36.854065 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.854047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "104d8cc4-e4c3-40be-849e-98707b1c8dcd" (UID: "104d8cc4-e4c3-40be-849e-98707b1c8dcd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:36.952678 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.952597    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:36.952678 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.952633    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:36.952678 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.952643    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrchx\" (UniqueName: \"kubernetes.io/projected/104d8cc4-e4c3-40be-849e-98707b1c8dcd-kube-api-access-lrchx\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:36.952678 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.952653    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:36.952678 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:36.952662    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/104d8cc4-e4c3-40be-849e-98707b1c8dcd-console-oauth-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:00:37.349697 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.349660    2563 generic.go:356] "Generic (PLEG): container finished" podID="104d8cc4-e4c3-40be-849e-98707b1c8dcd" containerID="e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2" exitCode=0
Sep 17 22:00:37.350139 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.349702    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d98c5dd95-mcgzd"
Sep 17 22:00:37.350139 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.349738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d98c5dd95-mcgzd" event={"ID":"104d8cc4-e4c3-40be-849e-98707b1c8dcd","Type":"ContainerDied","Data":"e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2"}
Sep 17 22:00:37.350139 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.349780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d98c5dd95-mcgzd" event={"ID":"104d8cc4-e4c3-40be-849e-98707b1c8dcd","Type":"ContainerDied","Data":"11e732f1b97ee155a794c79c6c1675e7d90fe13a77d32d8bb5d1bc57cee31d68"}
Sep 17 22:00:37.350139 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.349802    2563 scope.go:117] "RemoveContainer" containerID="e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2"
Sep 17 22:00:37.358072 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.358056    2563 scope.go:117] "RemoveContainer" containerID="e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2"
Sep 17 22:00:37.358365 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:00:37.358345    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2\": container with ID starting with e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2 not found: ID does not exist" containerID="e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2"
Sep 17 22:00:37.358447 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.358375    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2"} err="failed to get container status \"e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2\": rpc error: code = NotFound desc = could not find container \"e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2\": container with ID starting with e6629efaec8010c132644c9873eafc48e47d2a91cf339b1c06d8bb357bcc58b2 not found: ID does not exist"
Sep 17 22:00:37.369293 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.369270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d98c5dd95-mcgzd"]
Sep 17 22:00:37.373156 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.373137    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d98c5dd95-mcgzd"]
Sep 17 22:00:37.713819 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:00:37.713786    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104d8cc4-e4c3-40be-849e-98707b1c8dcd" path="/var/lib/kubelet/pods/104d8cc4-e4c3-40be-849e-98707b1c8dcd/volumes"
Sep 17 22:01:06.076837 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:01:06.076737    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:01:06.095675 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:01:06.095648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:01:06.482680 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:01:06.482653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:03:15.618296 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:03:15.618265    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:03:15.622763 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:03:15.622743    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:03:15.624273 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:03:15.624253    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 22:04:23.177194 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177164    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"]
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177434    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca563cf-0666-4ad1-8fce-b65ce76011de" containerName="console"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca563cf-0666-4ad1-8fce-b65ce76011de" containerName="console"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177451    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee37e1e-49fd-42ae-bcbd-942a1a2898e7" containerName="registry"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177456    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee37e1e-49fd-42ae-bcbd-942a1a2898e7" containerName="registry"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104d8cc4-e4c3-40be-849e-98707b1c8dcd" containerName="console"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177481    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="104d8cc4-e4c3-40be-849e-98707b1c8dcd" containerName="console"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177520    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee37e1e-49fd-42ae-bcbd-942a1a2898e7" containerName="registry"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177527    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="104d8cc4-e4c3-40be-849e-98707b1c8dcd" containerName="console"
Sep 17 22:04:23.177641 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.177534    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca563cf-0666-4ad1-8fce-b65ce76011de" containerName="console"
Sep 17 22:04:23.179274 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.179258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.182075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.182049    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:04:23.182229 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.182087    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 22:04:23.182229 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.182089    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:04:23.182721 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.182702    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-4b92x\""
Sep 17 22:04:23.189070 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.189048    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"]
Sep 17 22:04:23.336097 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.336061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2f6s\" (UniqueName: \"kubernetes.io/projected/dcafe906-ee50-4ca4-b4de-df43d3138008-kube-api-access-q2f6s\") pod \"llmisvc-controller-manager-645f9c8fd6-7wwqw\" (UID: \"dcafe906-ee50-4ca4-b4de-df43d3138008\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.336396 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.336376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcafe906-ee50-4ca4-b4de-df43d3138008-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7wwqw\" (UID: \"dcafe906-ee50-4ca4-b4de-df43d3138008\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.436813 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.436745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcafe906-ee50-4ca4-b4de-df43d3138008-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7wwqw\" (UID: \"dcafe906-ee50-4ca4-b4de-df43d3138008\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.436813 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.436799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2f6s\" (UniqueName: \"kubernetes.io/projected/dcafe906-ee50-4ca4-b4de-df43d3138008-kube-api-access-q2f6s\") pod \"llmisvc-controller-manager-645f9c8fd6-7wwqw\" (UID: \"dcafe906-ee50-4ca4-b4de-df43d3138008\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.439063 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.439042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcafe906-ee50-4ca4-b4de-df43d3138008-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7wwqw\" (UID: \"dcafe906-ee50-4ca4-b4de-df43d3138008\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.444962 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.444939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2f6s\" (UniqueName: \"kubernetes.io/projected/dcafe906-ee50-4ca4-b4de-df43d3138008-kube-api-access-q2f6s\") pod \"llmisvc-controller-manager-645f9c8fd6-7wwqw\" (UID: \"dcafe906-ee50-4ca4-b4de-df43d3138008\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.489936 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.489914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:23.760213 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.760193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"]
Sep 17 22:04:23.762826 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:04:23.762791    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddcafe906_ee50_4ca4_b4de_df43d3138008.slice/crio-91a79b193fc6f60bb43c72788b30afede82756809f46877bf11d2e76d9271c65 WatchSource:0}: Error finding container 91a79b193fc6f60bb43c72788b30afede82756809f46877bf11d2e76d9271c65: Status 404 returned error can't find the container with id 91a79b193fc6f60bb43c72788b30afede82756809f46877bf11d2e76d9271c65
Sep 17 22:04:23.764164 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:23.764146    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:04:24.166407 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:24.166323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw" event={"ID":"dcafe906-ee50-4ca4-b4de-df43d3138008","Type":"ContainerStarted","Data":"91a79b193fc6f60bb43c72788b30afede82756809f46877bf11d2e76d9271c65"}
Sep 17 22:04:27.179910 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:27.179878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw" event={"ID":"dcafe906-ee50-4ca4-b4de-df43d3138008","Type":"ContainerStarted","Data":"51b1fe4e80b3088eec6f7c45341992894f69b6dbbdf935daaa61dd7fef0c523f"}
Sep 17 22:04:27.180304 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:27.179940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:04:27.195480 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:27.195439    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw" podStartSLOduration=1.6914659090000002 podStartE2EDuration="4.195423997s" totalImagesPullingTime="2.503958088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:27.194570608 +0000 UTC m=+373.678056342" watchObservedRunningTime="2026-09-17 22:04:27.195423997 +0000 UTC m=+373.678909766"
Sep 17 22:04:58.188359 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:04:58.188332    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7wwqw"
Sep 17 22:05:57.026848 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.026812    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-xhmk4"]
Sep 17 22:05:57.029083 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.029068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.031467 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.031448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 22:05:57.031575 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.031453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 22:05:57.031575 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.031540    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z6gz2\""
Sep 17 22:05:57.037432 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.037408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xhmk4"]
Sep 17 22:05:57.162460 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.162418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9def44e-a618-4cfe-be84-6cb09323f157-model-cache\") pod \"s3-init-xhmk4\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") " pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.162460 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.162465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbds7\" (UniqueName: \"kubernetes.io/projected/a9def44e-a618-4cfe-be84-6cb09323f157-kube-api-access-zbds7\") pod \"s3-init-xhmk4\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") " pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.263769 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.263727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9def44e-a618-4cfe-be84-6cb09323f157-model-cache\") pod \"s3-init-xhmk4\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") " pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.263769 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.263774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbds7\" (UniqueName: \"kubernetes.io/projected/a9def44e-a618-4cfe-be84-6cb09323f157-kube-api-access-zbds7\") pod \"s3-init-xhmk4\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") " pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.264158 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.264140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9def44e-a618-4cfe-be84-6cb09323f157-model-cache\") pod \"s3-init-xhmk4\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") " pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.272089 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.272066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbds7\" (UniqueName: \"kubernetes.io/projected/a9def44e-a618-4cfe-be84-6cb09323f157-kube-api-access-zbds7\") pod \"s3-init-xhmk4\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") " pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.338867 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.338767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xhmk4"
Sep 17 22:05:57.606119 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:57.606030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xhmk4"]
Sep 17 22:05:57.609038 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:05:57.609007    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9def44e_a618_4cfe_be84_6cb09323f157.slice/crio-7a9e7ea0828efb83384d1532f40910ba3651d7dd7acc88dc99d310fcbc251547 WatchSource:0}: Error finding container 7a9e7ea0828efb83384d1532f40910ba3651d7dd7acc88dc99d310fcbc251547: Status 404 returned error can't find the container with id 7a9e7ea0828efb83384d1532f40910ba3651d7dd7acc88dc99d310fcbc251547
Sep 17 22:05:58.531843 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:05:58.531808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xhmk4" event={"ID":"a9def44e-a618-4cfe-be84-6cb09323f157","Type":"ContainerStarted","Data":"7a9e7ea0828efb83384d1532f40910ba3651d7dd7acc88dc99d310fcbc251547"}
Sep 17 22:06:01.545991 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:06:01.545949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xhmk4" event={"ID":"a9def44e-a618-4cfe-be84-6cb09323f157","Type":"ContainerStarted","Data":"908fba144be0db7bf85fedfc311cad623b797ea166dfc48c8525019c20c08179"}
Sep 17 22:06:02.550959 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:06:02.550924    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9def44e-a618-4cfe-be84-6cb09323f157" containerID="908fba144be0db7bf85fedfc311cad623b797ea166dfc48c8525019c20c08179" exitCode=0
Sep 17 22:06:02.551401 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:06:02.550980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xhmk4" event={"ID":"a9def44e-a618-4cfe-be84-6cb09323f157","Type":"ContainerDied","Data":"908fba144be0db7bf85fedfc311cad623b797ea166dfc48c8525019c20c08179"}
Sep 17 22:06:07.572983 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:06:07.572942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xhmk4" event={"ID":"a9def44e-a618-4cfe-be84-6cb09323f157","Type":"ContainerStarted","Data":"9b6140209b45a0abeb5ac6301625b2c5b94fafb7921b41182f1008626a458aaf"}
Sep 17 22:06:07.590735 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:06:07.590670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-xhmk4" podStartSLOduration=1.315079671 podStartE2EDuration="10.59065464s" totalImagesPullingTime="7.495188125s" totalInitContainerRuntime="1.780386844s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:07.589317172 +0000 UTC m=+474.072802904" watchObservedRunningTime="2026-09-17 22:06:07.59065464 +0000 UTC m=+474.074140373"
Sep 17 22:07:12.809632 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:12.809598    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9def44e-a618-4cfe-be84-6cb09323f157" containerID="9b6140209b45a0abeb5ac6301625b2c5b94fafb7921b41182f1008626a458aaf" exitCode=0
Sep 17 22:07:12.810033 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:12.809670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xhmk4" event={"ID":"a9def44e-a618-4cfe-be84-6cb09323f157","Type":"ContainerDied","Data":"9b6140209b45a0abeb5ac6301625b2c5b94fafb7921b41182f1008626a458aaf"}
Sep 17 22:07:13.934887 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:13.934864    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xhmk4"
Sep 17 22:07:13.970760 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:13.970732    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbds7\" (UniqueName: \"kubernetes.io/projected/a9def44e-a618-4cfe-be84-6cb09323f157-kube-api-access-zbds7\") pod \"a9def44e-a618-4cfe-be84-6cb09323f157\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") "
Sep 17 22:07:13.970904 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:13.970772    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9def44e-a618-4cfe-be84-6cb09323f157-model-cache\") pod \"a9def44e-a618-4cfe-be84-6cb09323f157\" (UID: \"a9def44e-a618-4cfe-be84-6cb09323f157\") "
Sep 17 22:07:13.972920 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:13.972896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9def44e-a618-4cfe-be84-6cb09323f157-kube-api-access-zbds7" (OuterVolumeSpecName: "kube-api-access-zbds7") pod "a9def44e-a618-4cfe-be84-6cb09323f157" (UID: "a9def44e-a618-4cfe-be84-6cb09323f157"). InnerVolumeSpecName "kube-api-access-zbds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:07:13.989716 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:13.989688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9def44e-a618-4cfe-be84-6cb09323f157-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9def44e-a618-4cfe-be84-6cb09323f157" (UID: "a9def44e-a618-4cfe-be84-6cb09323f157"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:14.071567 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:14.071488    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbds7\" (UniqueName: \"kubernetes.io/projected/a9def44e-a618-4cfe-be84-6cb09323f157-kube-api-access-zbds7\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:07:14.071567 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:14.071520    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9def44e-a618-4cfe-be84-6cb09323f157-model-cache\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:07:14.818596 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:14.818570    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xhmk4"
Sep 17 22:07:14.818596 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:14.818596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xhmk4" event={"ID":"a9def44e-a618-4cfe-be84-6cb09323f157","Type":"ContainerDied","Data":"7a9e7ea0828efb83384d1532f40910ba3651d7dd7acc88dc99d310fcbc251547"}
Sep 17 22:07:14.818798 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:14.818625    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9e7ea0828efb83384d1532f40910ba3651d7dd7acc88dc99d310fcbc251547"
Sep 17 22:07:26.747997 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.747959    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"]
Sep 17 22:07:26.748365 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.748277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9def44e-a618-4cfe-be84-6cb09323f157" containerName="s3-init"
Sep 17 22:07:26.748365 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.748290    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9def44e-a618-4cfe-be84-6cb09323f157" containerName="s3-init"
Sep 17 22:07:26.748365 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.748299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9def44e-a618-4cfe-be84-6cb09323f157" containerName="download-hf-model"
Sep 17 22:07:26.748365 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.748307    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9def44e-a618-4cfe-be84-6cb09323f157" containerName="download-hf-model"
Sep 17 22:07:26.748493 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.748367    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9def44e-a618-4cfe-be84-6cb09323f157" containerName="s3-init"
Sep 17 22:07:26.750581 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.750552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.750973 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.750943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk7ln\" (UniqueName: \"kubernetes.io/projected/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-api-access-qk7ln\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.751078 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.750987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.751078 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.751052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.751183 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.751085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.751183 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.751148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.753403 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.753385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:07:26.753403 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.753398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:07:26.753545 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.753398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:07:26.753545 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.753429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:07:26.754248 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.754229    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 22:07:26.754335 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.754246    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 22:07:26.754532 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.754518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:07:26.762183 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.762163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"]
Sep 17 22:07:26.851537 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.851511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.851680 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.851544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.851680 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.851571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.851680 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.851605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk7ln\" (UniqueName: \"kubernetes.io/projected/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-api-access-qk7ln\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.851680 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.851645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.851999 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.851978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.852213 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.852195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.852268 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.852249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.853952 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.853934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.873733 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.873711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk7ln\" (UniqueName: \"kubernetes.io/projected/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-api-access-qk7ln\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-psslt\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:26.936909 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.936882    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"]
Sep 17 22:07:26.939502 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.939481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:26.943128 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.943096    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 22:07:26.952459 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.952438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcs6x\" (UniqueName: \"kubernetes.io/projected/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-api-access-xcs6x\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:26.952548 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.952465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:26.952548 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.952492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:26.952627 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.952551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:26.952627 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.952566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:26.959677 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:26.959654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"]
Sep 17 22:07:27.053606 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.053533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.053606 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.053563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.053606 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.053600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs6x\" (UniqueName: \"kubernetes.io/projected/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-api-access-xcs6x\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.053868 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.053616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.053868 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.053646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.053868 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:07:27.053697    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 22:07:27.053868 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:07:27.053778    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls podName:c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06 nodeName:}" failed. No retries permitted until 2026-09-17 22:07:27.553757861 +0000 UTC m=+554.037243587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls") pod "isvc-canary-rollback-predictor-76cbc5464-gdmmc" (UID: "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 22:07:27.054094 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.054075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.054307 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.054291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.054344 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.054308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.060661 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.060641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:27.075638 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.075607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs6x\" (UniqueName: \"kubernetes.io/projected/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-api-access-xcs6x\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.332283 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.332210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"]
Sep 17 22:07:27.335173 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:07:27.335148    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96aac6a_bdb1_4f0e_b6b4_36c68f4e9546.slice/crio-7ead18cb02ebdda071e900c5ba19f9ce60fbf7a08c731d44c8092001efe5ec7b WatchSource:0}: Error finding container 7ead18cb02ebdda071e900c5ba19f9ce60fbf7a08c731d44c8092001efe5ec7b: Status 404 returned error can't find the container with id 7ead18cb02ebdda071e900c5ba19f9ce60fbf7a08c731d44c8092001efe5ec7b
Sep 17 22:07:27.557069 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.557030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.559355 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.559334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-gdmmc\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.850066 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.850026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:27.875732 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.875687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerStarted","Data":"1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96"}
Sep 17 22:07:27.875858 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:27.875776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerStarted","Data":"7ead18cb02ebdda071e900c5ba19f9ce60fbf7a08c731d44c8092001efe5ec7b"}
Sep 17 22:07:28.122910 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:28.122883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"]
Sep 17 22:07:28.125334 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:07:28.125306    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9acfdff_0a7b_4c55_98dd_0dbbc4b5fe06.slice/crio-1049843c1387ec1da4590adf11b5ef12e494b0b5f47601c00a08cbcc2e4e323e WatchSource:0}: Error finding container 1049843c1387ec1da4590adf11b5ef12e494b0b5f47601c00a08cbcc2e4e323e: Status 404 returned error can't find the container with id 1049843c1387ec1da4590adf11b5ef12e494b0b5f47601c00a08cbcc2e4e323e
Sep 17 22:07:28.881495 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:28.881452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerStarted","Data":"fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158"}
Sep 17 22:07:28.881937 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:28.881514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerStarted","Data":"1049843c1387ec1da4590adf11b5ef12e494b0b5f47601c00a08cbcc2e4e323e"}
Sep 17 22:07:31.895360 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:31.895321    2563 generic.go:356] "Generic (PLEG): container finished" podID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerID="1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96" exitCode=0
Sep 17 22:07:31.895702 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:31.895389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerDied","Data":"1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96"}
Sep 17 22:07:32.903740 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:32.903705    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerID="fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158" exitCode=0
Sep 17 22:07:32.904202 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:32.903801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerDied","Data":"fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158"}
Sep 17 22:07:47.993273 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:47.993223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerStarted","Data":"dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98"}
Sep 17 22:07:47.995342 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:47.995305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerStarted","Data":"5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09"}
Sep 17 22:07:51.011590 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.011543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerStarted","Data":"0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae"}
Sep 17 22:07:51.012075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.011625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:51.012075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.011733    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:51.013197 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.013167    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:07:51.013818 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.013798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerStarted","Data":"2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a"}
Sep 17 22:07:51.013954 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.013937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:51.030950 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.030905    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podStartSLOduration=2.95324565 podStartE2EDuration="25.030889162s" totalImagesPullingTime="18.146435183s" totalInitContainerRuntime="3.931208329s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:51.029874695 +0000 UTC m=+577.513360429" watchObservedRunningTime="2026-09-17 22:07:51.030889162 +0000 UTC m=+577.514374896"
Sep 17 22:07:51.048475 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:51.048417    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podStartSLOduration=4.055230695 podStartE2EDuration="25.048403174s" totalImagesPullingTime="17.124572046s" totalInitContainerRuntime="3.868600433s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:51.047480803 +0000 UTC m=+577.530966539" watchObservedRunningTime="2026-09-17 22:07:51.048403174 +0000 UTC m=+577.531888908"
Sep 17 22:07:52.018093 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:52.018050    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:07:52.018536 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:52.018271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:52.019371 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:52.019350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:07:53.022788 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:53.022751    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:07:57.022318 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:57.022293    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:07:57.022841 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:57.022817    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:07:58.027336 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:58.027306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:07:58.027817 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:07:58.027791    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:08:07.023210 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:07.023168    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:08:08.027942 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:08.027898    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:08:15.649377 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:15.649349    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:08:15.651283 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:15.651254    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:08:17.023670 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:17.023633    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:08:18.028401 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:18.028358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:08:27.022922 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:27.022877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:08:28.027927 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:28.027851    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:08:37.022766 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:37.022726    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:08:38.027706 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:38.027666    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:08:47.023359 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:47.023321    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:08:48.028243 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:48.028206    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:08:57.023398 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:57.023360    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:08:58.027961 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:08:58.027926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 22:09:07.024084 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:07.024049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:09:07.142278 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:07.142228    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"]
Sep 17 22:09:07.142670 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:07.142618    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container" containerID="cri-o://dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98" gracePeriod=30
Sep 17 22:09:07.143689 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:07.143014    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kube-rbac-proxy" containerID="cri-o://2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a" gracePeriod=30
Sep 17 22:09:07.143689 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:07.143294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:09:07.316453 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:07.316370    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerID="2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a" exitCode=2
Sep 17 22:09:07.316589 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:07.316454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerDied","Data":"2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a"}
Sep 17 22:09:08.023289 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:08.023252    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 22:09:11.482652 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.482619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:09:11.620728 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.620629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcs6x\" (UniqueName: \"kubernetes.io/projected/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-api-access-xcs6x\") pod \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") "
Sep 17 22:09:11.620728 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.620712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls\") pod \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") "
Sep 17 22:09:11.620728 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.620730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-cabundle-cert\") pod \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") "
Sep 17 22:09:11.620991 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.620760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kserve-provision-location\") pod \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") "
Sep 17 22:09:11.620991 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.620777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-rbac-proxy-sar-config\") pod \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\" (UID: \"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06\") "
Sep 17 22:09:11.621176 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.621146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" (UID: "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:11.621231 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.621150    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" (UID: "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:11.621231 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.621201    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" (UID: "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:11.622892 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.622871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" (UID: "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:09:11.622980 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.622960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-api-access-xcs6x" (OuterVolumeSpecName: "kube-api-access-xcs6x") pod "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" (UID: "c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06"). InnerVolumeSpecName "kube-api-access-xcs6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:11.721770 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.721750    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:11.721770 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.721770    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:11.721910 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.721783    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcs6x\" (UniqueName: \"kubernetes.io/projected/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-kube-api-access-xcs6x\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:11.721910 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.721792    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:11.721910 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:11.721801    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:12.338006 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.337974    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerID="dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98" exitCode=0
Sep 17 22:09:12.338192 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.338028    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"
Sep 17 22:09:12.338192 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.338044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerDied","Data":"dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98"}
Sep 17 22:09:12.338192 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.338092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc" event={"ID":"c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06","Type":"ContainerDied","Data":"1049843c1387ec1da4590adf11b5ef12e494b0b5f47601c00a08cbcc2e4e323e"}
Sep 17 22:09:12.338192 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.338136    2563 scope.go:117] "RemoveContainer" containerID="2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a"
Sep 17 22:09:12.345978 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.345964    2563 scope.go:117] "RemoveContainer" containerID="dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98"
Sep 17 22:09:12.353125 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.353093    2563 scope.go:117] "RemoveContainer" containerID="fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158"
Sep 17 22:09:12.359316 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.359147    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"]
Sep 17 22:09:12.363035 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.362460    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-gdmmc"]
Sep 17 22:09:12.365008 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.364904    2563 scope.go:117] "RemoveContainer" containerID="2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a"
Sep 17 22:09:12.365190 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:09:12.365171    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a\": container with ID starting with 2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a not found: ID does not exist" containerID="2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a"
Sep 17 22:09:12.365234 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.365199    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a"} err="failed to get container status \"2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a\": rpc error: code = NotFound desc = could not find container \"2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a\": container with ID starting with 2569c3d007ac44d2f17e63cb9c5b5b57620d575f315eed6fdb9eb25276d5bb0a not found: ID does not exist"
Sep 17 22:09:12.365234 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.365216    2563 scope.go:117] "RemoveContainer" containerID="dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98"
Sep 17 22:09:12.365478 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:09:12.365458    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98\": container with ID starting with dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98 not found: ID does not exist" containerID="dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98"
Sep 17 22:09:12.365519 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.365485    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98"} err="failed to get container status \"dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98\": rpc error: code = NotFound desc = could not find container \"dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98\": container with ID starting with dead9ac1381468b2d9b59a3c904d24dac6cb1e33e82268a816f70ad23a816c98 not found: ID does not exist"
Sep 17 22:09:12.365519 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.365500    2563 scope.go:117] "RemoveContainer" containerID="fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158"
Sep 17 22:09:12.365726 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:09:12.365708    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158\": container with ID starting with fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158 not found: ID does not exist" containerID="fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158"
Sep 17 22:09:12.365780 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:12.365734    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158"} err="failed to get container status \"fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158\": rpc error: code = NotFound desc = could not find container \"fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158\": container with ID starting with fb234523eb96ed79558ca189f68ad61affbe0582d683e0ea5f22dcc12be11158 not found: ID does not exist"
Sep 17 22:09:13.710087 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:13.710054    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" path="/var/lib/kubelet/pods/c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06/volumes"
Sep 17 22:09:16.643908 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.643877    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"]
Sep 17 22:09:16.644287 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.644135    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" containerID="cri-o://5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09" gracePeriod=30
Sep 17 22:09:16.644287 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.644166    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kube-rbac-proxy" containerID="cri-o://0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae" gracePeriod=30
Sep 17 22:09:16.709365 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709337    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"]
Sep 17 22:09:16.709632 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709620    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="storage-initializer"
Sep 17 22:09:16.709683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709634    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="storage-initializer"
Sep 17 22:09:16.709683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709643    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container"
Sep 17 22:09:16.709683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709648    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container"
Sep 17 22:09:16.709683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709661    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kube-rbac-proxy"
Sep 17 22:09:16.709683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kube-rbac-proxy"
Sep 17 22:09:16.709845 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709718    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kube-rbac-proxy"
Sep 17 22:09:16.709845 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.709728    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9acfdff-0a7b-4c55-98dd-0dbbc4b5fe06" containerName="kserve-container"
Sep 17 22:09:16.712776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.712762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.715080 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.715060    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 22:09:16.726232 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.726210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"]
Sep 17 22:09:16.759396 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.759378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.759455 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.759411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.759505 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.759480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/281c7c9f-904a-4b34-adfb-03c3eb543302-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.759557 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.759540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p24d\" (UniqueName: \"kubernetes.io/projected/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-api-access-6p24d\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.759649 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.759572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/281c7c9f-904a-4b34-adfb-03c3eb543302-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.860756 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.860725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/281c7c9f-904a-4b34-adfb-03c3eb543302-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.860902 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.860769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p24d\" (UniqueName: \"kubernetes.io/projected/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-api-access-6p24d\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.860902 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.860887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/281c7c9f-904a-4b34-adfb-03c3eb543302-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.861020 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.860972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.861074 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.861025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.861277 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.861253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/281c7c9f-904a-4b34-adfb-03c3eb543302-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.861571 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.861551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.861643 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.861618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.863009 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.862993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/281c7c9f-904a-4b34-adfb-03c3eb543302-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:16.869117 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:16.869090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p24d\" (UniqueName: \"kubernetes.io/projected/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-api-access-6p24d\") pod \"isvc-canary-rollback-predictor-76cbc5464-tr8ds\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:17.019146 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.019088    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 22:09:17.022306 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.022286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:17.023471 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.023437    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 22:09:17.292950 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.292868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"]
Sep 17 22:09:17.295844 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:09:17.295816    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281c7c9f_904a_4b34_adfb_03c3eb543302.slice/crio-85fe54461b74cec8dcfadfdd090ddea84724f03ea59dd1adc11cdc28fbd426ef WatchSource:0}: Error finding container 85fe54461b74cec8dcfadfdd090ddea84724f03ea59dd1adc11cdc28fbd426ef: Status 404 returned error can't find the container with id 85fe54461b74cec8dcfadfdd090ddea84724f03ea59dd1adc11cdc28fbd426ef
Sep 17 22:09:17.360502 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.360467    2563 generic.go:356] "Generic (PLEG): container finished" podID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerID="0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae" exitCode=2
Sep 17 22:09:17.360627 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.360516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerDied","Data":"0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae"}
Sep 17 22:09:17.362243 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.362214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerStarted","Data":"cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca"}
Sep 17 22:09:17.362348 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:17.362276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerStarted","Data":"85fe54461b74cec8dcfadfdd090ddea84724f03ea59dd1adc11cdc28fbd426ef"}
Sep 17 22:09:20.983882 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.983859    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:09:20.991687 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.991669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-proxy-tls\") pod \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") "
Sep 17 22:09:20.991791 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.991700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk7ln\" (UniqueName: \"kubernetes.io/projected/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-api-access-qk7ln\") pod \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") "
Sep 17 22:09:20.991791 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.991721    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kserve-provision-location\") pod \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") "
Sep 17 22:09:20.991791 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.991759    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-rbac-proxy-sar-config\") pod \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") "
Sep 17 22:09:20.991948 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.991805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-cabundle-cert\") pod \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\" (UID: \"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546\") "
Sep 17 22:09:20.992157 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.992127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" (UID: "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:20.992157 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.992148    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" (UID: "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:20.992320 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.992181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" (UID: "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:20.993683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.993659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" (UID: "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:09:20.993775 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:20.993728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-api-access-qk7ln" (OuterVolumeSpecName: "kube-api-access-qk7ln") pod "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" (UID: "f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546"). InnerVolumeSpecName "kube-api-access-qk7ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:21.092552 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.092481    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:21.092552 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.092506    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:21.092552 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.092519    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:21.092552 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.092530    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk7ln\" (UniqueName: \"kubernetes.io/projected/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kube-api-access-qk7ln\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:21.092552 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.092541    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:09:21.381359 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.381271    2563 generic.go:356] "Generic (PLEG): container finished" podID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerID="5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09" exitCode=0
Sep 17 22:09:21.381359 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.381336    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"
Sep 17 22:09:21.381359 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.381338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerDied","Data":"5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09"}
Sep 17 22:09:21.381581 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.381401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt" event={"ID":"f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546","Type":"ContainerDied","Data":"7ead18cb02ebdda071e900c5ba19f9ce60fbf7a08c731d44c8092001efe5ec7b"}
Sep 17 22:09:21.381581 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.381439    2563 scope.go:117] "RemoveContainer" containerID="0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae"
Sep 17 22:09:21.383129 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.383089    2563 generic.go:356] "Generic (PLEG): container finished" podID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerID="cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca" exitCode=0
Sep 17 22:09:21.383251 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.383161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerDied","Data":"cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca"}
Sep 17 22:09:21.390248 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.390234    2563 scope.go:117] "RemoveContainer" containerID="5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09"
Sep 17 22:09:21.397733 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.397712    2563 scope.go:117] "RemoveContainer" containerID="1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96"
Sep 17 22:09:21.406390 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.406373    2563 scope.go:117] "RemoveContainer" containerID="0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae"
Sep 17 22:09:21.406635 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:09:21.406616    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae\": container with ID starting with 0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae not found: ID does not exist" containerID="0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae"
Sep 17 22:09:21.406747 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.406645    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae"} err="failed to get container status \"0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae\": rpc error: code = NotFound desc = could not find container \"0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae\": container with ID starting with 0b67ee3c3b92ca0fd1621165ab426302a9a9a2eec81c523ddc2c17e52221deae not found: ID does not exist"
Sep 17 22:09:21.406747 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.406668    2563 scope.go:117] "RemoveContainer" containerID="5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09"
Sep 17 22:09:21.406950 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:09:21.406922    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09\": container with ID starting with 5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09 not found: ID does not exist" containerID="5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09"
Sep 17 22:09:21.407043 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.406958    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09"} err="failed to get container status \"5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09\": rpc error: code = NotFound desc = could not find container \"5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09\": container with ID starting with 5cf54f264881751dd9c81c78fb905e8448a1a25ca092c3b640784acb0b37ce09 not found: ID does not exist"
Sep 17 22:09:21.407043 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.406978    2563 scope.go:117] "RemoveContainer" containerID="1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96"
Sep 17 22:09:21.407247 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:09:21.407229    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96\": container with ID starting with 1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96 not found: ID does not exist" containerID="1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96"
Sep 17 22:09:21.407326 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.407249    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96"} err="failed to get container status \"1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96\": rpc error: code = NotFound desc = could not find container \"1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96\": container with ID starting with 1f8fe1365e6580d4af7a06729d511524795854def6a65b00c3d5defad4bbfe96 not found: ID does not exist"
Sep 17 22:09:21.419305 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.419285    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"]
Sep 17 22:09:21.422533 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.422515    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-psslt"]
Sep 17 22:09:21.710856 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:21.710822    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" path="/var/lib/kubelet/pods/f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546/volumes"
Sep 17 22:09:22.392203 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:22.392173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerStarted","Data":"8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de"}
Sep 17 22:09:22.392652 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:22.392233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerStarted","Data":"883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29"}
Sep 17 22:09:22.392652 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:22.392515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:22.392771 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:22.392677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:22.393745 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:22.393722    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:09:22.411926 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:22.411860    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podStartSLOduration=2.592515944 podStartE2EDuration="6.411846407s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.819330463s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:22.410940298 +0000 UTC m=+668.894426043" watchObservedRunningTime="2026-09-17 22:09:22.411846407 +0000 UTC m=+668.895332141"
Sep 17 22:09:23.395765 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:23.395723    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:09:28.400205 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:28.400178    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:09:28.400716 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:28.400687    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:09:38.401219 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:38.401183    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:09:48.400881 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:48.400839    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:09:58.400875 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:09:58.400799    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:10:08.400891 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:08.400845    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:10:18.401308 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:18.401264    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:10:28.400622 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:28.400581    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:10:38.401182 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:38.401152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:10:46.790317 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:46.790285    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"]
Sep 17 22:10:46.791039 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:46.790553    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" containerID="cri-o://883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29" gracePeriod=30
Sep 17 22:10:46.791039 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:46.790613    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kube-rbac-proxy" containerID="cri-o://8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de" gracePeriod=30
Sep 17 22:10:47.722149 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:47.722097    2563 generic.go:356] "Generic (PLEG): container finished" podID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerID="8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de" exitCode=2
Sep 17 22:10:47.722357 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:47.722169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerDied","Data":"8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de"}
Sep 17 22:10:48.396318 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:48.396273    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 17 22:10:48.400604 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:48.400580    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 22:10:51.226306 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.226280    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:10:51.282840 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.282813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-rbac-proxy-sar-config\") pod \"281c7c9f-904a-4b34-adfb-03c3eb543302\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") "
Sep 17 22:10:51.282979 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.282847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p24d\" (UniqueName: \"kubernetes.io/projected/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-api-access-6p24d\") pod \"281c7c9f-904a-4b34-adfb-03c3eb543302\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") "
Sep 17 22:10:51.282979 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.282871    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/281c7c9f-904a-4b34-adfb-03c3eb543302-kserve-provision-location\") pod \"281c7c9f-904a-4b34-adfb-03c3eb543302\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") "
Sep 17 22:10:51.282979 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.282892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-cabundle-cert\") pod \"281c7c9f-904a-4b34-adfb-03c3eb543302\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") "
Sep 17 22:10:51.282979 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.282943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/281c7c9f-904a-4b34-adfb-03c3eb543302-proxy-tls\") pod \"281c7c9f-904a-4b34-adfb-03c3eb543302\" (UID: \"281c7c9f-904a-4b34-adfb-03c3eb543302\") "
Sep 17 22:10:51.283217 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.283191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "281c7c9f-904a-4b34-adfb-03c3eb543302" (UID: "281c7c9f-904a-4b34-adfb-03c3eb543302"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:51.283260 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.283238    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281c7c9f-904a-4b34-adfb-03c3eb543302-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "281c7c9f-904a-4b34-adfb-03c3eb543302" (UID: "281c7c9f-904a-4b34-adfb-03c3eb543302"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:51.283364 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.283342    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "281c7c9f-904a-4b34-adfb-03c3eb543302" (UID: "281c7c9f-904a-4b34-adfb-03c3eb543302"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:51.284964 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.284944    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-api-access-6p24d" (OuterVolumeSpecName: "kube-api-access-6p24d") pod "281c7c9f-904a-4b34-adfb-03c3eb543302" (UID: "281c7c9f-904a-4b34-adfb-03c3eb543302"). InnerVolumeSpecName "kube-api-access-6p24d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:51.284964 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.284957    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281c7c9f-904a-4b34-adfb-03c3eb543302-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "281c7c9f-904a-4b34-adfb-03c3eb543302" (UID: "281c7c9f-904a-4b34-adfb-03c3eb543302"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:10:51.384489 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.384428    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:10:51.384489 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.384452    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p24d\" (UniqueName: \"kubernetes.io/projected/281c7c9f-904a-4b34-adfb-03c3eb543302-kube-api-access-6p24d\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:10:51.384489 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.384461    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/281c7c9f-904a-4b34-adfb-03c3eb543302-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:10:51.384489 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.384471    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/281c7c9f-904a-4b34-adfb-03c3eb543302-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:10:51.384489 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.384480    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/281c7c9f-904a-4b34-adfb-03c3eb543302-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:10:51.738490 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.738459    2563 generic.go:356] "Generic (PLEG): container finished" podID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerID="883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29" exitCode=0
Sep 17 22:10:51.738617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.738520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerDied","Data":"883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29"}
Sep 17 22:10:51.738617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.738562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds" event={"ID":"281c7c9f-904a-4b34-adfb-03c3eb543302","Type":"ContainerDied","Data":"85fe54461b74cec8dcfadfdd090ddea84724f03ea59dd1adc11cdc28fbd426ef"}
Sep 17 22:10:51.738617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.738592    2563 scope.go:117] "RemoveContainer" containerID="8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de"
Sep 17 22:10:51.738617 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.738518    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"
Sep 17 22:10:51.746578 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.746556    2563 scope.go:117] "RemoveContainer" containerID="883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29"
Sep 17 22:10:51.753337 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.753322    2563 scope.go:117] "RemoveContainer" containerID="cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca"
Sep 17 22:10:51.756016 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.755989    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"]
Sep 17 22:10:51.759683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.759662    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-tr8ds"]
Sep 17 22:10:51.760531 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.760518    2563 scope.go:117] "RemoveContainer" containerID="8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de"
Sep 17 22:10:51.760803 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:10:51.760787    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de\": container with ID starting with 8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de not found: ID does not exist" containerID="8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de"
Sep 17 22:10:51.760872 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.760814    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de"} err="failed to get container status \"8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de\": rpc error: code = NotFound desc = could not find container \"8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de\": container with ID starting with 8c2ab6fbbacb7fbe51877deb16d20f3419f1cfbcd6c047a145186e5f417852de not found: ID does not exist"
Sep 17 22:10:51.760872 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.760836    2563 scope.go:117] "RemoveContainer" containerID="883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29"
Sep 17 22:10:51.761078 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:10:51.761059    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29\": container with ID starting with 883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29 not found: ID does not exist" containerID="883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29"
Sep 17 22:10:51.761148 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.761085    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29"} err="failed to get container status \"883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29\": rpc error: code = NotFound desc = could not find container \"883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29\": container with ID starting with 883bc6f6f2d2553935ed6eef313c4b58dd66f4b2909f6178263db8305630bd29 not found: ID does not exist"
Sep 17 22:10:51.761148 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.761129    2563 scope.go:117] "RemoveContainer" containerID="cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca"
Sep 17 22:10:51.761402 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:10:51.761387    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca\": container with ID starting with cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca not found: ID does not exist" containerID="cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca"
Sep 17 22:10:51.761440 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:51.761407    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca"} err="failed to get container status \"cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca\": rpc error: code = NotFound desc = could not find container \"cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca\": container with ID starting with cb8cbd0471897cea0086156d6aa15534164ec020384d822db10941db3bfc09ca not found: ID does not exist"
Sep 17 22:10:52.059237 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059161    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"]
Sep 17 22:10:52.059497 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059480    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="storage-initializer"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="storage-initializer"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059507    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kube-rbac-proxy"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059513    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kube-rbac-proxy"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="storage-initializer"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="storage-initializer"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container"
Sep 17 22:10:52.059547 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059549    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kube-rbac-proxy"
Sep 17 22:10:52.059776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059554    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kube-rbac-proxy"
Sep 17 22:10:52.059776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059568    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container"
Sep 17 22:10:52.059776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059573    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container"
Sep 17 22:10:52.059776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059617    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kserve-container"
Sep 17 22:10:52.059776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kserve-container"
Sep 17 22:10:52.059776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059632    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" containerName="kube-rbac-proxy"
Sep 17 22:10:52.059776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.059639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96aac6a-bdb1-4f0e-b6b4-36c68f4e9546" containerName="kube-rbac-proxy"
Sep 17 22:10:52.062549 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.062528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.066438 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.066418    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 22:10:52.070126 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.070094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:10:52.070403 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.070387    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:10:52.070458 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.070410    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:10:52.070458 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.070438    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 22:10:52.070556 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.070537    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:10:52.076386 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.076373    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:10:52.103168 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.103144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"]
Sep 17 22:10:52.191833 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.191802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.192004 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.191870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74nrg\" (UniqueName: \"kubernetes.io/projected/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-api-access-74nrg\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.192004 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.191896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.192004 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.191924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.192004 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.191973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293144 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293548 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293548 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293548 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74nrg\" (UniqueName: \"kubernetes.io/projected/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-api-access-74nrg\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293548 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293548 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:10:52.293335    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 22:10:52.293548 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:10:52.293423    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls podName:37f4c7e8-4d10-482a-bd59-2e8e09fde07c nodeName:}" failed. No retries permitted until 2026-09-17 22:10:52.793404807 +0000 UTC m=+759.276890533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls") pod "isvc-canary-stop-predictor-646f5bdbcb-4pghx" (UID: "37f4c7e8-4d10-482a-bd59-2e8e09fde07c") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 22:10:52.293758 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293848 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.293884 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.293849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.301523 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.301503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74nrg\" (UniqueName: \"kubernetes.io/projected/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-api-access-74nrg\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.797023 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.796989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.799380 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.799362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-4pghx\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:52.972462 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:52.972421    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:53.242360 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:53.242335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"]
Sep 17 22:10:53.244828 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:10:53.244803    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f4c7e8_4d10_482a_bd59_2e8e09fde07c.slice/crio-9dbb55d2c10b9a9498b9ac6a823490715ea905e3089fac9be9394ff2bfcdeb60 WatchSource:0}: Error finding container 9dbb55d2c10b9a9498b9ac6a823490715ea905e3089fac9be9394ff2bfcdeb60: Status 404 returned error can't find the container with id 9dbb55d2c10b9a9498b9ac6a823490715ea905e3089fac9be9394ff2bfcdeb60
Sep 17 22:10:53.710500 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:53.710459    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281c7c9f-904a-4b34-adfb-03c3eb543302" path="/var/lib/kubelet/pods/281c7c9f-904a-4b34-adfb-03c3eb543302/volumes"
Sep 17 22:10:53.749157 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:53.749130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerStarted","Data":"cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d"}
Sep 17 22:10:53.749300 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:53.749182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerStarted","Data":"9dbb55d2c10b9a9498b9ac6a823490715ea905e3089fac9be9394ff2bfcdeb60"}
Sep 17 22:10:57.764865 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:57.764831    2563 generic.go:356] "Generic (PLEG): container finished" podID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerID="cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d" exitCode=0
Sep 17 22:10:57.765369 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:57.764869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerDied","Data":"cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d"}
Sep 17 22:10:58.770568 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:58.770534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerStarted","Data":"aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693"}
Sep 17 22:10:58.770936 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:58.770596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerStarted","Data":"ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51"}
Sep 17 22:10:58.771992 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:58.771963    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:58.772141 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:58.772060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:10:58.775973 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:58.775930    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:10:58.792968 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:58.792924    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podStartSLOduration=2.86337524 podStartE2EDuration="6.792912138s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.929536898s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:58.791363932 +0000 UTC m=+765.274849666" watchObservedRunningTime="2026-09-17 22:10:58.792912138 +0000 UTC m=+765.276397871"
Sep 17 22:10:59.774777 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:10:59.774731    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:11:04.779893 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:04.779864    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:11:04.780493 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:04.780469    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:11:12.283091 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.283057    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"]
Sep 17 22:11:12.286528 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.286509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.289058 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.289021    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-5ac84-predictor-serving-cert\""
Sep 17 22:11:12.289058 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.289039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-5ac84-kube-rbac-proxy-sar-config\""
Sep 17 22:11:12.297502 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.297478    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"]
Sep 17 22:11:12.456277 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.456239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.456465 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.456291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.456465 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.456351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.456465 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.456375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.456465 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.456449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x6q\" (UniqueName: \"kubernetes.io/projected/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-api-access-74x6q\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.556991 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.556908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.556991 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.556945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.556991 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.556972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.557289 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.557008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x6q\" (UniqueName: \"kubernetes.io/projected/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-api-access-74x6q\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.557289 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.557050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.557289 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:11:12.557060    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-5ac84-predictor-serving-cert" not found
Sep 17 22:11:12.557289 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:11:12.557148    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls podName:6369b675-fe7c-4b62-99a3-c09f4c12ce27 nodeName:}" failed. No retries permitted until 2026-09-17 22:11:13.057129091 +0000 UTC m=+779.540614803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" (UID: "6369b675-fe7c-4b62-99a3-c09f4c12ce27") : secret "isvc-xgboost-graph-raw-hpa-5ac84-predictor-serving-cert" not found
Sep 17 22:11:12.557506 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.557351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.557677 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.557658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.557746 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.557663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:12.565612 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:12.565590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x6q\" (UniqueName: \"kubernetes.io/projected/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-api-access-74x6q\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:13.059559 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:13.059522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:13.062016 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:13.061985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:13.197083 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:13.197053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:13.467897 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:13.467866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"]
Sep 17 22:11:13.470349 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:11:13.470314    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6369b675_fe7c_4b62_99a3_c09f4c12ce27.slice/crio-de1bec34cdd1beb8d887b5c1f1b76bb19d86a0f0b7946dd8d23a00891c9de1c2 WatchSource:0}: Error finding container de1bec34cdd1beb8d887b5c1f1b76bb19d86a0f0b7946dd8d23a00891c9de1c2: Status 404 returned error can't find the container with id de1bec34cdd1beb8d887b5c1f1b76bb19d86a0f0b7946dd8d23a00891c9de1c2
Sep 17 22:11:13.830878 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:13.830800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerStarted","Data":"88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede"}
Sep 17 22:11:13.830878 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:13.830860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerStarted","Data":"de1bec34cdd1beb8d887b5c1f1b76bb19d86a0f0b7946dd8d23a00891c9de1c2"}
Sep 17 22:11:14.780369 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:14.780333    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:11:17.847927 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:17.847841    2563 generic.go:356] "Generic (PLEG): container finished" podID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerID="88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede" exitCode=0
Sep 17 22:11:17.847927 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:17.847913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerDied","Data":"88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede"}
Sep 17 22:11:17.849316 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:17.849298    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:11:24.780969 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:24.780930    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:11:34.781409 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:34.781317    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:11:37.941260 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:37.941221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerStarted","Data":"f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac"}
Sep 17 22:11:37.941687 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:37.941286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerStarted","Data":"fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509"}
Sep 17 22:11:37.941687 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:37.941566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:37.941760 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:37.941737    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:37.942883 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:37.942857    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:11:37.960802 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:37.960750    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podStartSLOduration=2.412400934 podStartE2EDuration="25.960734638s" totalImagesPullingTime="19.556876372s" totalInitContainerRuntime="3.991457332s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:11:37.959140785 +0000 UTC m=+804.442626516" watchObservedRunningTime="2026-09-17 22:11:37.960734638 +0000 UTC m=+804.444220435"
Sep 17 22:11:38.946238 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:38.946189    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:11:39.950426 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:39.950382    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:11:44.780690 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:44.780651    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:11:44.955897 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:44.955871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:11:44.956394 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:44.956349    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:11:54.781337 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:54.781297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:11:54.956942 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:11:54.956892    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:12:04.780776 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:04.780730    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:12:04.956546 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:04.956504    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:12:06.705555 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:06.705514    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:12:14.956653 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:14.956618    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:12:16.707047 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:16.707016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:12:23.304940 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:23.304905    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"]
Sep 17 22:12:23.305517 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:23.305252    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" containerID="cri-o://ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51" gracePeriod=30
Sep 17 22:12:23.305517 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:23.305325    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kube-rbac-proxy" containerID="cri-o://aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693" gracePeriod=30
Sep 17 22:12:24.132001 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:24.131969    2563 generic.go:356] "Generic (PLEG): container finished" podID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerID="aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693" exitCode=2
Sep 17 22:12:24.132190 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:24.132019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerDied","Data":"aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693"}
Sep 17 22:12:24.775691 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:24.775652    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 22:12:24.956543 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:24.956505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:12:26.706434 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:26.706395    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 22:12:27.741830 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.741804    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:12:27.772693 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.772667    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-rbac-proxy-sar-config\") pod \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") "
Sep 17 22:12:27.772801 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.772711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls\") pod \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") "
Sep 17 22:12:27.772801 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.772750    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kserve-provision-location\") pod \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") "
Sep 17 22:12:27.772801 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.772777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-cabundle-cert\") pod \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") "
Sep 17 22:12:27.772963 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.772820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74nrg\" (UniqueName: \"kubernetes.io/projected/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-api-access-74nrg\") pod \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\" (UID: \"37f4c7e8-4d10-482a-bd59-2e8e09fde07c\") "
Sep 17 22:12:27.773134 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.773087    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37f4c7e8-4d10-482a-bd59-2e8e09fde07c" (UID: "37f4c7e8-4d10-482a-bd59-2e8e09fde07c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:27.773206 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.773121    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37f4c7e8-4d10-482a-bd59-2e8e09fde07c" (UID: "37f4c7e8-4d10-482a-bd59-2e8e09fde07c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:27.773206 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.773147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37f4c7e8-4d10-482a-bd59-2e8e09fde07c" (UID: "37f4c7e8-4d10-482a-bd59-2e8e09fde07c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:27.774857 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.774835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-api-access-74nrg" (OuterVolumeSpecName: "kube-api-access-74nrg") pod "37f4c7e8-4d10-482a-bd59-2e8e09fde07c" (UID: "37f4c7e8-4d10-482a-bd59-2e8e09fde07c"). InnerVolumeSpecName "kube-api-access-74nrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:12:27.774935 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.774876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37f4c7e8-4d10-482a-bd59-2e8e09fde07c" (UID: "37f4c7e8-4d10-482a-bd59-2e8e09fde07c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:12:27.874090 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.874019    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:12:27.874090 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.874045    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:12:27.874090 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.874059    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:12:27.874090 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.874071    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:12:27.874090 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:27.874083    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74nrg\" (UniqueName: \"kubernetes.io/projected/37f4c7e8-4d10-482a-bd59-2e8e09fde07c-kube-api-access-74nrg\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:12:28.148834 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.148741    2563 generic.go:356] "Generic (PLEG): container finished" podID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerID="ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51" exitCode=0
Sep 17 22:12:28.148834 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.148791    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"
Sep 17 22:12:28.149021 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.148833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerDied","Data":"ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51"}
Sep 17 22:12:28.149021 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.148894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx" event={"ID":"37f4c7e8-4d10-482a-bd59-2e8e09fde07c","Type":"ContainerDied","Data":"9dbb55d2c10b9a9498b9ac6a823490715ea905e3089fac9be9394ff2bfcdeb60"}
Sep 17 22:12:28.149021 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.148922    2563 scope.go:117] "RemoveContainer" containerID="aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693"
Sep 17 22:12:28.160233 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.160206    2563 scope.go:117] "RemoveContainer" containerID="ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51"
Sep 17 22:12:28.168086 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.168069    2563 scope.go:117] "RemoveContainer" containerID="cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d"
Sep 17 22:12:28.172823 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.172799    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"]
Sep 17 22:12:28.175395 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.175370    2563 scope.go:117] "RemoveContainer" containerID="aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693"
Sep 17 22:12:28.175674 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:12:28.175653    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693\": container with ID starting with aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693 not found: ID does not exist" containerID="aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693"
Sep 17 22:12:28.175741 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.175681    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693"} err="failed to get container status \"aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693\": rpc error: code = NotFound desc = could not find container \"aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693\": container with ID starting with aa61910358efb77c7ab6143175d0372572c89220cdd6b041cb5c076c74f76693 not found: ID does not exist"
Sep 17 22:12:28.175741 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.175697    2563 scope.go:117] "RemoveContainer" containerID="ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51"
Sep 17 22:12:28.175987 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:12:28.175970    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51\": container with ID starting with ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51 not found: ID does not exist" containerID="ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51"
Sep 17 22:12:28.176047 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.175996    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51"} err="failed to get container status \"ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51\": rpc error: code = NotFound desc = could not find container \"ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51\": container with ID starting with ac2a0c8accf63a7ef0f422dcd7372af6a2624a81ecaed607a40319d23f582c51 not found: ID does not exist"
Sep 17 22:12:28.176047 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.176019    2563 scope.go:117] "RemoveContainer" containerID="cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d"
Sep 17 22:12:28.176356 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:12:28.176334    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d\": container with ID starting with cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d not found: ID does not exist" containerID="cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d"
Sep 17 22:12:28.176407 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.176365    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d"} err="failed to get container status \"cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d\": rpc error: code = NotFound desc = could not find container \"cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d\": container with ID starting with cacab792e7533e16c939cd1c89d70d04621461aa22001e0a633e18b84ba6916d not found: ID does not exist"
Sep 17 22:12:28.176864 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:28.176847    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-4pghx"]
Sep 17 22:12:29.710503 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:29.710473    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" path="/var/lib/kubelet/pods/37f4c7e8-4d10-482a-bd59-2e8e09fde07c/volumes"
Sep 17 22:12:34.957122 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:34.957062    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:12:44.957228 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:12:44.957196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:13:02.373845 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:02.373769    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"]
Sep 17 22:13:02.374335 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:02.374091    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" containerID="cri-o://fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509" gracePeriod=30
Sep 17 22:13:02.374335 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:02.374149    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kube-rbac-proxy" containerID="cri-o://f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac" gracePeriod=30
Sep 17 22:13:03.286352 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:03.286317    2563 generic.go:356] "Generic (PLEG): container finished" podID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerID="f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac" exitCode=2
Sep 17 22:13:03.286507 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:03.286361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerDied","Data":"f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac"}
Sep 17 22:13:04.951242 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:04.951190    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 22:13:04.956836 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:04.956807    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 22:13:06.213491 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.213462    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:13:06.299949 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.299915    2563 generic.go:356] "Generic (PLEG): container finished" podID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerID="fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509" exitCode=0
Sep 17 22:13:06.300142 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.299979    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"
Sep 17 22:13:06.300142 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.299996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerDied","Data":"fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509"}
Sep 17 22:13:06.300142 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.300069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s" event={"ID":"6369b675-fe7c-4b62-99a3-c09f4c12ce27","Type":"ContainerDied","Data":"de1bec34cdd1beb8d887b5c1f1b76bb19d86a0f0b7946dd8d23a00891c9de1c2"}
Sep 17 22:13:06.300142 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.300128    2563 scope.go:117] "RemoveContainer" containerID="f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac"
Sep 17 22:13:06.307798 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.307781    2563 scope.go:117] "RemoveContainer" containerID="fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509"
Sep 17 22:13:06.314835 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.314821    2563 scope.go:117] "RemoveContainer" containerID="88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede"
Sep 17 22:13:06.321801 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.321782    2563 scope.go:117] "RemoveContainer" containerID="f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac"
Sep 17 22:13:06.322081 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:13:06.322063    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac\": container with ID starting with f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac not found: ID does not exist" containerID="f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac"
Sep 17 22:13:06.322167 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.322094    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac"} err="failed to get container status \"f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac\": rpc error: code = NotFound desc = could not find container \"f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac\": container with ID starting with f07370a1da0649eb9634a778e94f4600c7d9ba33a20e9048c4a75453fc4f24ac not found: ID does not exist"
Sep 17 22:13:06.322167 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.322138    2563 scope.go:117] "RemoveContainer" containerID="fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509"
Sep 17 22:13:06.322396 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:13:06.322379    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509\": container with ID starting with fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509 not found: ID does not exist" containerID="fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509"
Sep 17 22:13:06.322438 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.322402    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509"} err="failed to get container status \"fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509\": rpc error: code = NotFound desc = could not find container \"fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509\": container with ID starting with fdbe5c0aade4c28c8c48206cbdef0fcd0219b175668b40e9750cc1d1958db509 not found: ID does not exist"
Sep 17 22:13:06.322438 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.322421    2563 scope.go:117] "RemoveContainer" containerID="88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede"
Sep 17 22:13:06.322654 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:13:06.322637    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede\": container with ID starting with 88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede not found: ID does not exist" containerID="88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede"
Sep 17 22:13:06.322693 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.322659    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede"} err="failed to get container status \"88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede\": rpc error: code = NotFound desc = could not find container \"88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede\": container with ID starting with 88f85df767617956576acbdf2acf43c9acf3c7eb90f9ebb392a8e92ad6abbede not found: ID does not exist"
Sep 17 22:13:06.372061 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-cabundle-cert\") pod \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") "
Sep 17 22:13:06.372159 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kserve-provision-location\") pod \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") "
Sep 17 22:13:06.372159 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372122    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls\") pod \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") "
Sep 17 22:13:06.372159 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74x6q\" (UniqueName: \"kubernetes.io/projected/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-api-access-74x6q\") pod \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") "
Sep 17 22:13:06.372303 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372168    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-rbac-proxy-sar-config\") pod \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\" (UID: \"6369b675-fe7c-4b62-99a3-c09f4c12ce27\") "
Sep 17 22:13:06.372483 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372460    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6369b675-fe7c-4b62-99a3-c09f4c12ce27" (UID: "6369b675-fe7c-4b62-99a3-c09f4c12ce27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:06.372483 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6369b675-fe7c-4b62-99a3-c09f4c12ce27" (UID: "6369b675-fe7c-4b62-99a3-c09f4c12ce27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:13:06.372575 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.372545    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6369b675-fe7c-4b62-99a3-c09f4c12ce27" (UID: "6369b675-fe7c-4b62-99a3-c09f4c12ce27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:13:06.374173 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.374153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6369b675-fe7c-4b62-99a3-c09f4c12ce27" (UID: "6369b675-fe7c-4b62-99a3-c09f4c12ce27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:13:06.374251 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.374175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-api-access-74x6q" (OuterVolumeSpecName: "kube-api-access-74x6q") pod "6369b675-fe7c-4b62-99a3-c09f4c12ce27" (UID: "6369b675-fe7c-4b62-99a3-c09f4c12ce27"). InnerVolumeSpecName "kube-api-access-74x6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:13:06.473619 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.473587    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:13:06.473619 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.473612    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6369b675-fe7c-4b62-99a3-c09f4c12ce27-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:13:06.473619 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.473624    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74x6q\" (UniqueName: \"kubernetes.io/projected/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-api-access-74x6q\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:13:06.473891 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.473632    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:13:06.473891 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.473640    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6369b675-fe7c-4b62-99a3-c09f4c12ce27-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:13:06.622654 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.622626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"]
Sep 17 22:13:06.626381 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:06.626353    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5ac84-predictor-658c549f4f-28t6s"]
Sep 17 22:13:07.710485 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:07.710447    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" path="/var/lib/kubelet/pods/6369b675-fe7c-4b62-99a3-c09f4c12ce27/volumes"
Sep 17 22:13:15.676750 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:15.676725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:13:15.677216 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:13:15.677126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.548288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"]
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.548953    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="storage-initializer"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.548974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="storage-initializer"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.548998    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549034    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549059    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="storage-initializer"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549068    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="storage-initializer"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549088    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kube-rbac-proxy"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549097    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kube-rbac-proxy"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549138    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kube-rbac-proxy"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549147    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kube-rbac-proxy"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549283    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kube-rbac-proxy"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549303    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kserve-container"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549322    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f4c7e8-4d10-482a-bd59-2e8e09fde07c" containerName="kserve-container"
Sep 17 22:14:32.551221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.549332    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6369b675-fe7c-4b62-99a3-c09f4c12ce27" containerName="kube-rbac-proxy"
Sep 17 22:14:32.553294 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.553268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.555867 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.555845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:14:32.556088 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.556071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ba0165-predictor-serving-cert\""
Sep 17 22:14:32.556164 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.556139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ba0165\""
Sep 17 22:14:32.556548 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.556531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:14:32.556847 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.556832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ba0165-dockercfg-ntstp\""
Sep 17 22:14:32.556894 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.556876    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ba0165-kube-rbac-proxy-sar-config\""
Sep 17 22:14:32.556973 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.556955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:14:32.578030 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.578004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"]
Sep 17 22:14:32.618410 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.618385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.618514 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.618426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f558r\" (UniqueName: \"kubernetes.io/projected/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-api-access-f558r\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.618514 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.618455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-proxy-tls\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.618598 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.618529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kserve-provision-location\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.618598 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.618566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-cabundle-cert\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.719648 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.719609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-cabundle-cert\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.719832 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.719659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.719832 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.719691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f558r\" (UniqueName: \"kubernetes.io/projected/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-api-access-f558r\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.719832 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.719709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-proxy-tls\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.719832 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.719737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kserve-provision-location\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.720209 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.720176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kserve-provision-location\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.720405 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.720387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-cabundle-cert\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.720485 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.720441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.722193 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.722176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-proxy-tls\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.730465 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.730445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f558r\" (UniqueName: \"kubernetes.io/projected/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-api-access-f558r\") pod \"isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") " pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:32.863308 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:32.863216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:33.135442 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:33.135415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"]
Sep 17 22:14:33.137725 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:14:33.137698    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b9f0fa_9eb2_482d_82cb_d64df4453f07.slice/crio-c66d582a802665f0f6f58a8d0e7df00047b5d2cbb424cd292be800c27fbd3457 WatchSource:0}: Error finding container c66d582a802665f0f6f58a8d0e7df00047b5d2cbb424cd292be800c27fbd3457: Status 404 returned error can't find the container with id c66d582a802665f0f6f58a8d0e7df00047b5d2cbb424cd292be800c27fbd3457
Sep 17 22:14:33.631195 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:33.631158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w" event={"ID":"c4b9f0fa-9eb2-482d-82cb-d64df4453f07","Type":"ContainerStarted","Data":"a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699"}
Sep 17 22:14:33.631589 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:33.631215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w" event={"ID":"c4b9f0fa-9eb2-482d-82cb-d64df4453f07","Type":"ContainerStarted","Data":"c66d582a802665f0f6f58a8d0e7df00047b5d2cbb424cd292be800c27fbd3457"}
Sep 17 22:14:38.653414 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:38.653384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_c4b9f0fa-9eb2-482d-82cb-d64df4453f07/storage-initializer/0.log"
Sep 17 22:14:38.653868 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:38.653421    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerID="a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699" exitCode=1
Sep 17 22:14:38.653868 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:38.653502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w" event={"ID":"c4b9f0fa-9eb2-482d-82cb-d64df4453f07","Type":"ContainerDied","Data":"a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699"}
Sep 17 22:14:39.659566 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:39.659537    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_c4b9f0fa-9eb2-482d-82cb-d64df4453f07/storage-initializer/0.log"
Sep 17 22:14:39.659968 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:39.659647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w" event={"ID":"c4b9f0fa-9eb2-482d-82cb-d64df4453f07","Type":"ContainerStarted","Data":"a4909d1912fdceb08c6402f78f7eaa6614951d6fea50271b960bb0dff735827c"}
Sep 17 22:14:44.680088 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:44.680061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_c4b9f0fa-9eb2-482d-82cb-d64df4453f07/storage-initializer/1.log"
Sep 17 22:14:44.680463 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:44.680424    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_c4b9f0fa-9eb2-482d-82cb-d64df4453f07/storage-initializer/0.log"
Sep 17 22:14:44.680463 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:44.680452    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerID="a4909d1912fdceb08c6402f78f7eaa6614951d6fea50271b960bb0dff735827c" exitCode=1
Sep 17 22:14:44.680554 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:44.680530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w" event={"ID":"c4b9f0fa-9eb2-482d-82cb-d64df4453f07","Type":"ContainerDied","Data":"a4909d1912fdceb08c6402f78f7eaa6614951d6fea50271b960bb0dff735827c"}
Sep 17 22:14:44.680648 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:44.680632    2563 scope.go:117] "RemoveContainer" containerID="a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699"
Sep 17 22:14:44.680983 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:44.680965    2563 scope.go:117] "RemoveContainer" containerID="a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699"
Sep 17 22:14:44.690566 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:14:44.690543    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_kserve-ci-e2e-test(c4b9f0fa-9eb2-482d-82cb-d64df4453f07)\"" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07"
Sep 17 22:14:44.690804 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:14:44.690774    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_kserve-ci-e2e-test_c4b9f0fa-9eb2-482d-82cb-d64df4453f07_0 in pod sandbox c66d582a802665f0f6f58a8d0e7df00047b5d2cbb424cd292be800c27fbd3457 from index: no such id: 'a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699'" containerID="a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699"
Sep 17 22:14:44.690891 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:44.690807    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_kserve-ci-e2e-test_c4b9f0fa-9eb2-482d-82cb-d64df4453f07_0 in pod sandbox c66d582a802665f0f6f58a8d0e7df00047b5d2cbb424cd292be800c27fbd3457 from index: no such id: 'a849dc4c0bd7ce35b4f09ff7a09cff34374729400f608004fb0a1eed82d5e699'"
Sep 17 22:14:45.685688 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:45.685657    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_c4b9f0fa-9eb2-482d-82cb-d64df4453f07/storage-initializer/1.log"
Sep 17 22:14:50.547985 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.547954    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"]
Sep 17 22:14:50.670328 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.670304    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_c4b9f0fa-9eb2-482d-82cb-d64df4453f07/storage-initializer/1.log"
Sep 17 22:14:50.674211 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.674194    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:50.704689 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.704670    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w_c4b9f0fa-9eb2-482d-82cb-d64df4453f07/storage-initializer/1.log"
Sep 17 22:14:50.704788 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.704754    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"
Sep 17 22:14:50.704856 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.704780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w" event={"ID":"c4b9f0fa-9eb2-482d-82cb-d64df4453f07","Type":"ContainerDied","Data":"c66d582a802665f0f6f58a8d0e7df00047b5d2cbb424cd292be800c27fbd3457"}
Sep 17 22:14:50.704913 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.704849    2563 scope.go:117] "RemoveContainer" containerID="a4909d1912fdceb08c6402f78f7eaa6614951d6fea50271b960bb0dff735827c"
Sep 17 22:14:50.864879 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.864814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f558r\" (UniqueName: \"kubernetes.io/projected/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-api-access-f558r\") pod \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") "
Sep 17 22:14:50.864879 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.864868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-rbac-proxy-sar-config\") pod \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") "
Sep 17 22:14:50.865048 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.864894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kserve-provision-location\") pod \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") "
Sep 17 22:14:50.865048 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.864916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-proxy-tls\") pod \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") "
Sep 17 22:14:50.865048 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.864944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-cabundle-cert\") pod \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\" (UID: \"c4b9f0fa-9eb2-482d-82cb-d64df4453f07\") "
Sep 17 22:14:50.865255 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.865224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4b9f0fa-9eb2-482d-82cb-d64df4453f07" (UID: "c4b9f0fa-9eb2-482d-82cb-d64df4453f07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:50.865315 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.865274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4b9f0fa-9eb2-482d-82cb-d64df4453f07" (UID: "c4b9f0fa-9eb2-482d-82cb-d64df4453f07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:50.865408 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.865380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4b9f0fa-9eb2-482d-82cb-d64df4453f07" (UID: "c4b9f0fa-9eb2-482d-82cb-d64df4453f07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:50.866935 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.866915    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-api-access-f558r" (OuterVolumeSpecName: "kube-api-access-f558r") pod "c4b9f0fa-9eb2-482d-82cb-d64df4453f07" (UID: "c4b9f0fa-9eb2-482d-82cb-d64df4453f07"). InnerVolumeSpecName "kube-api-access-f558r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:50.867033 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.866987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4b9f0fa-9eb2-482d-82cb-d64df4453f07" (UID: "c4b9f0fa-9eb2-482d-82cb-d64df4453f07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:50.965487 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.965458    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f558r\" (UniqueName: \"kubernetes.io/projected/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-api-access-f558r\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:14:50.965487 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.965484    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:14:50.965639 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.965494    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:14:50.965639 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.965503    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:14:50.965639 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:50.965514    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b9f0fa-9eb2-482d-82cb-d64df4453f07-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:14:51.048271 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:51.048242    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"]
Sep 17 22:14:51.053683 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:51.053659    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba0165-predictor-c46cc96dc-2tg4w"]
Sep 17 22:14:51.710317 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:51.710288    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" path="/var/lib/kubelet/pods/c4b9f0fa-9eb2-482d-82cb-d64df4453f07/volumes"
Sep 17 22:14:55.739294 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.739263    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"]
Sep 17 22:14:55.739644 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.739542    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerName="storage-initializer"
Sep 17 22:14:55.739644 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.739556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerName="storage-initializer"
Sep 17 22:14:55.739644 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.739615    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerName="storage-initializer"
Sep 17 22:14:55.739644 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.739623    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerName="storage-initializer"
Sep 17 22:14:55.739774 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.739669    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerName="storage-initializer"
Sep 17 22:14:55.739774 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.739676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b9f0fa-9eb2-482d-82cb-d64df4453f07" containerName="storage-initializer"
Sep 17 22:14:55.743969 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.743954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:55.746410 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.746390    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-254f1c\""
Sep 17 22:14:55.746784 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.746766    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:14:55.746865 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.746809    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-254f1c-kube-rbac-proxy-sar-config\""
Sep 17 22:14:55.746865 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.746848    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:14:55.746975 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.746848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-254f1c-predictor-serving-cert\""
Sep 17 22:14:55.747406 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.747391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-254f1c-dockercfg-2r8mm\""
Sep 17 22:14:55.747485 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.747420    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:14:55.754544 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.754522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"]
Sep 17 22:14:55.899545 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.899516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhm6c\" (UniqueName: \"kubernetes.io/projected/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-api-access-vhm6c\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:55.899657 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.899560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:55.899657 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.899640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b28d65-76c2-4bdb-9647-59f7ee395089-proxy-tls\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:55.899748 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.899663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b28d65-76c2-4bdb-9647-59f7ee395089-kserve-provision-location\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:55.899748 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:55.899690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-cabundle-cert\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.000305 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.000247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.000305 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.000288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b28d65-76c2-4bdb-9647-59f7ee395089-proxy-tls\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.000466 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.000308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b28d65-76c2-4bdb-9647-59f7ee395089-kserve-provision-location\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.000466 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.000344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-cabundle-cert\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.000466 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.000385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhm6c\" (UniqueName: \"kubernetes.io/projected/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-api-access-vhm6c\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.000732 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.000711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b28d65-76c2-4bdb-9647-59f7ee395089-kserve-provision-location\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.000889 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.000870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.001062 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.001042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-cabundle-cert\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.002836 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.002815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b28d65-76c2-4bdb-9647-59f7ee395089-proxy-tls\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.007806 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.007781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhm6c\" (UniqueName: \"kubernetes.io/projected/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-api-access-vhm6c\") pod \"isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") " pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.053631 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.053610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:14:56.324351 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.324264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"]
Sep 17 22:14:56.326534 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:14:56.326508    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b28d65_76c2_4bdb_9647_59f7ee395089.slice/crio-68757775ec722ffdeefcc81d1a8da27f72dd09686abd01a386f519cbd2b5cd70 WatchSource:0}: Error finding container 68757775ec722ffdeefcc81d1a8da27f72dd09686abd01a386f519cbd2b5cd70: Status 404 returned error can't find the container with id 68757775ec722ffdeefcc81d1a8da27f72dd09686abd01a386f519cbd2b5cd70
Sep 17 22:14:56.729413 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.729381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p" event={"ID":"50b28d65-76c2-4bdb-9647-59f7ee395089","Type":"ContainerStarted","Data":"69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22"}
Sep 17 22:14:56.729576 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:14:56.729444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p" event={"ID":"50b28d65-76c2-4bdb-9647-59f7ee395089","Type":"ContainerStarted","Data":"68757775ec722ffdeefcc81d1a8da27f72dd09686abd01a386f519cbd2b5cd70"}
Sep 17 22:15:02.753837 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:02.753811    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p_50b28d65-76c2-4bdb-9647-59f7ee395089/storage-initializer/0.log"
Sep 17 22:15:02.754255 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:02.753848    2563 generic.go:356] "Generic (PLEG): container finished" podID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerID="69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22" exitCode=1
Sep 17 22:15:02.754255 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:02.753921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p" event={"ID":"50b28d65-76c2-4bdb-9647-59f7ee395089","Type":"ContainerDied","Data":"69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22"}
Sep 17 22:15:03.759652 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:03.759624    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p_50b28d65-76c2-4bdb-9647-59f7ee395089/storage-initializer/0.log"
Sep 17 22:15:03.760047 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:03.759705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p" event={"ID":"50b28d65-76c2-4bdb-9647-59f7ee395089","Type":"ContainerStarted","Data":"1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba"}
Sep 17 22:15:05.738491 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:05.738454    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"]
Sep 17 22:15:05.738876 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:05.738708    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerName="storage-initializer" containerID="cri-o://1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba" gracePeriod=30
Sep 17 22:15:06.082963 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.082936    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p_50b28d65-76c2-4bdb-9647-59f7ee395089/storage-initializer/1.log"
Sep 17 22:15:06.083382 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.083365    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p_50b28d65-76c2-4bdb-9647-59f7ee395089/storage-initializer/0.log"
Sep 17 22:15:06.085577 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.085555    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:15:06.180704 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.180664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-cabundle-cert\") pod \"50b28d65-76c2-4bdb-9647-59f7ee395089\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") "
Sep 17 22:15:06.180704 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.180713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b28d65-76c2-4bdb-9647-59f7ee395089-kserve-provision-location\") pod \"50b28d65-76c2-4bdb-9647-59f7ee395089\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") "
Sep 17 22:15:06.180921 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.180740    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b28d65-76c2-4bdb-9647-59f7ee395089-proxy-tls\") pod \"50b28d65-76c2-4bdb-9647-59f7ee395089\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") "
Sep 17 22:15:06.180921 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.180766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhm6c\" (UniqueName: \"kubernetes.io/projected/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-api-access-vhm6c\") pod \"50b28d65-76c2-4bdb-9647-59f7ee395089\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") "
Sep 17 22:15:06.180921 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.180801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-rbac-proxy-sar-config\") pod \"50b28d65-76c2-4bdb-9647-59f7ee395089\" (UID: \"50b28d65-76c2-4bdb-9647-59f7ee395089\") "
Sep 17 22:15:06.181032 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.180987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b28d65-76c2-4bdb-9647-59f7ee395089-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50b28d65-76c2-4bdb-9647-59f7ee395089" (UID: "50b28d65-76c2-4bdb-9647-59f7ee395089"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:06.181210 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.181184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50b28d65-76c2-4bdb-9647-59f7ee395089" (UID: "50b28d65-76c2-4bdb-9647-59f7ee395089"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:15:06.181328 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.181220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50b28d65-76c2-4bdb-9647-59f7ee395089" (UID: "50b28d65-76c2-4bdb-9647-59f7ee395089"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:15:06.182865 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.182842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-api-access-vhm6c" (OuterVolumeSpecName: "kube-api-access-vhm6c") pod "50b28d65-76c2-4bdb-9647-59f7ee395089" (UID: "50b28d65-76c2-4bdb-9647-59f7ee395089"). InnerVolumeSpecName "kube-api-access-vhm6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:15:06.182953 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.182911    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b28d65-76c2-4bdb-9647-59f7ee395089-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50b28d65-76c2-4bdb-9647-59f7ee395089" (UID: "50b28d65-76c2-4bdb-9647-59f7ee395089"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:15:06.282075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.281983    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b28d65-76c2-4bdb-9647-59f7ee395089-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:15:06.282075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.282014    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b28d65-76c2-4bdb-9647-59f7ee395089-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:15:06.282075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.282026    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhm6c\" (UniqueName: \"kubernetes.io/projected/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-api-access-vhm6c\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:15:06.282075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.282034    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:15:06.282075 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.282042    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b28d65-76c2-4bdb-9647-59f7ee395089-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:15:06.774572 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.774545    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p_50b28d65-76c2-4bdb-9647-59f7ee395089/storage-initializer/1.log"
Sep 17 22:15:06.775008 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.774973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p_50b28d65-76c2-4bdb-9647-59f7ee395089/storage-initializer/0.log"
Sep 17 22:15:06.775080 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.775007    2563 generic.go:356] "Generic (PLEG): container finished" podID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerID="1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba" exitCode=1
Sep 17 22:15:06.775169 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.775125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p" event={"ID":"50b28d65-76c2-4bdb-9647-59f7ee395089","Type":"ContainerDied","Data":"1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba"}
Sep 17 22:15:06.775169 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.775132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"
Sep 17 22:15:06.775278 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.775206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p" event={"ID":"50b28d65-76c2-4bdb-9647-59f7ee395089","Type":"ContainerDied","Data":"68757775ec722ffdeefcc81d1a8da27f72dd09686abd01a386f519cbd2b5cd70"}
Sep 17 22:15:06.775329 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.775243    2563 scope.go:117] "RemoveContainer" containerID="1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba"
Sep 17 22:15:06.784192 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.784171    2563 scope.go:117] "RemoveContainer" containerID="69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22"
Sep 17 22:15:06.792303 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.792277    2563 scope.go:117] "RemoveContainer" containerID="1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba"
Sep 17 22:15:06.792868 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:15:06.792841    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba\": container with ID starting with 1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba not found: ID does not exist" containerID="1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba"
Sep 17 22:15:06.792970 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.792878    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba"} err="failed to get container status \"1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba\": rpc error: code = NotFound desc = could not find container \"1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba\": container with ID starting with 1a378955e36a63203db51f7041987b36e77cf290c09d2eb3971d39ae03d70aba not found: ID does not exist"
Sep 17 22:15:06.792970 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.792902    2563 scope.go:117] "RemoveContainer" containerID="69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22"
Sep 17 22:15:06.793239 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:15:06.793217    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22\": container with ID starting with 69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22 not found: ID does not exist" containerID="69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22"
Sep 17 22:15:06.793333 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.793246    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22"} err="failed to get container status \"69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22\": rpc error: code = NotFound desc = could not find container \"69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22\": container with ID starting with 69f68b7c788055255662d7d08cf49b571207e3f6be12ba8bd554f075da103d22 not found: ID does not exist"
Sep 17 22:15:06.811737 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.811691    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"]
Sep 17 22:15:06.815420 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:06.815391    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-254f1c-predictor-5496fbb84c-8c48p"]
Sep 17 22:15:07.711377 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:15:07.711344    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" path="/var/lib/kubelet/pods/50b28d65-76c2-4bdb-9647-59f7ee395089/volumes"
Sep 17 22:16:46.311458 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.311427    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"]
Sep 17 22:16:46.311843 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.311731    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerName="storage-initializer"
Sep 17 22:16:46.311843 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.311742    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerName="storage-initializer"
Sep 17 22:16:46.311843 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.311754    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerName="storage-initializer"
Sep 17 22:16:46.311843 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.311760    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerName="storage-initializer"
Sep 17 22:16:46.311843 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.311808    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerName="storage-initializer"
Sep 17 22:16:46.311843 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.311817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50b28d65-76c2-4bdb-9647-59f7ee395089" containerName="storage-initializer"
Sep 17 22:16:46.314745 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.314729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.317192 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.317172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:16:46.317584 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.317559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:16:46.317584 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.317577    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:16:46.317584 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.317560    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:16:46.317772 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.317567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-eacee-predictor-serving-cert\""
Sep 17 22:16:46.318013 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.317995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-eacee-kube-rbac-proxy-sar-config\""
Sep 17 22:16:46.318120 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.318015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:16:46.325187 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.325168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"]
Sep 17 22:16:46.436815 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.436785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.436948 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.436833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppxc\" (UniqueName: \"kubernetes.io/projected/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-api-access-jppxc\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.436948 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.436917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.437064 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.436952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kserve-provision-location\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.437064 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.436984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-cabundle-cert\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538005 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.537974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538005 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.538011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kserve-provision-location\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538277 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.538036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-cabundle-cert\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538277 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.538083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538277 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.538132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppxc\" (UniqueName: \"kubernetes.io/projected/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-api-access-jppxc\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538494 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:16:46.538282    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-serving-cert: secret "raw-sklearn-runtime-eacee-predictor-serving-cert" not found
Sep 17 22:16:46.538494 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:16:46.538366    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls podName:38380e0d-f633-48d4-90f0-fcadd0f86fd0 nodeName:}" failed. No retries permitted until 2026-09-17 22:16:47.038345852 +0000 UTC m=+1113.521831565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls") pod "raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" (UID: "38380e0d-f633-48d4-90f0-fcadd0f86fd0") : secret "raw-sklearn-runtime-eacee-predictor-serving-cert" not found
Sep 17 22:16:46.538494 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.538473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kserve-provision-location\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538908 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.538887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.538944 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.538887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-cabundle-cert\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:46.546885 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:46.546865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppxc\" (UniqueName: \"kubernetes.io/projected/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-api-access-jppxc\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:47.042793 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:47.042763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:47.045088 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:47.045068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls\") pod \"raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:47.226450 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:47.226415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:47.498339 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:47.498312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"]
Sep 17 22:16:47.499953 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:16:47.499925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38380e0d_f633_48d4_90f0_fcadd0f86fd0.slice/crio-9b23d396eef379f66518ee09661a498576953eeb24d8baee5761bf423daedbab WatchSource:0}: Error finding container 9b23d396eef379f66518ee09661a498576953eeb24d8baee5761bf423daedbab: Status 404 returned error can't find the container with id 9b23d396eef379f66518ee09661a498576953eeb24d8baee5761bf423daedbab
Sep 17 22:16:48.148090 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:48.148058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerStarted","Data":"54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7"}
Sep 17 22:16:48.148279 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:48.148135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerStarted","Data":"9b23d396eef379f66518ee09661a498576953eeb24d8baee5761bf423daedbab"}
Sep 17 22:16:52.165312 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:52.165281    2563 generic.go:356] "Generic (PLEG): container finished" podID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerID="54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7" exitCode=0
Sep 17 22:16:52.165819 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:52.165352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerDied","Data":"54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7"}
Sep 17 22:16:53.171359 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:53.171324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerStarted","Data":"b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b"}
Sep 17 22:16:53.171750 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:53.171389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerStarted","Data":"84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71"}
Sep 17 22:16:53.171750 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:53.171578    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:53.193772 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:53.193728    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podStartSLOduration=3.274767339 podStartE2EDuration="7.193713992s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.918946653s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:53.191918732 +0000 UTC m=+1119.675404465" watchObservedRunningTime="2026-09-17 22:16:53.193713992 +0000 UTC m=+1119.677199726"
Sep 17 22:16:54.175619 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:54.175584    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:16:54.176731 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:54.176703    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:16:55.179056 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:16:55.179017    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:17:00.184027 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:00.183998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:17:00.184512 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:00.184486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:17:10.184630 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:10.184591    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:17:20.184833 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:20.184789    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:17:30.184533 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:30.184454    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:17:40.185190 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:40.185146    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:17:50.184904 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:50.184866    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:17:53.191334 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.191301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"]
Sep 17 22:17:53.194577 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.194561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.197207 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.197185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 22:17:53.197325 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.197259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 22:17:53.206513 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.206492    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"]
Sep 17 22:17:53.241713 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.241688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f123cc3a-f638-4be8-a686-69c4dacd8715-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.241852 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.241724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.241852 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.241756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.241852 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.241776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f123cc3a-f638-4be8-a686-69c4dacd8715-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.241852 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.241807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhtz\" (UniqueName: \"kubernetes.io/projected/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-api-access-flhtz\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342140 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342316 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f123cc3a-f638-4be8-a686-69c4dacd8715-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342316 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhtz\" (UniqueName: \"kubernetes.io/projected/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-api-access-flhtz\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342316 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f123cc3a-f638-4be8-a686-69c4dacd8715-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342316 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342544 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f123cc3a-f638-4be8-a686-69c4dacd8715-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342829 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.342915 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.342864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.344736 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.344718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f123cc3a-f638-4be8-a686-69c4dacd8715-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.349774 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.349756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhtz\" (UniqueName: \"kubernetes.io/projected/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-api-access-flhtz\") pod \"isvc-canary-create-predictor-5f845bd94d-4vtvn\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.504238 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.504201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:17:53.776345 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:53.776308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"]
Sep 17 22:17:53.779005 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:17:53.778979    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf123cc3a_f638_4be8_a686_69c4dacd8715.slice/crio-43ab2671101949c2734b87be0ec07241f60d59fd05a5bd0b2a9d57c58308b704 WatchSource:0}: Error finding container 43ab2671101949c2734b87be0ec07241f60d59fd05a5bd0b2a9d57c58308b704: Status 404 returned error can't find the container with id 43ab2671101949c2734b87be0ec07241f60d59fd05a5bd0b2a9d57c58308b704
Sep 17 22:17:54.402494 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:54.402460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerStarted","Data":"c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1"}
Sep 17 22:17:54.402855 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:54.402516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerStarted","Data":"43ab2671101949c2734b87be0ec07241f60d59fd05a5bd0b2a9d57c58308b704"}
Sep 17 22:17:58.420283 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:58.420247    2563 generic.go:356] "Generic (PLEG): container finished" podID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerID="c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1" exitCode=0
Sep 17 22:17:58.420716 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:58.420319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerDied","Data":"c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1"}
Sep 17 22:17:59.426646 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:59.426608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerStarted","Data":"78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1"}
Sep 17 22:17:59.426990 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:59.426701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerStarted","Data":"9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81"}
Sep 17 22:17:59.426990 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:17:59.426881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:18:00.184878 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:00.184838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:18:00.430600 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:00.430568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:18:00.431664 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:00.431637    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:18:01.434128 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:01.434073    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:18:06.438344 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:06.438317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:18:06.438837 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:06.438808    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:18:06.458935 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:06.458623    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podStartSLOduration=9.629744244 podStartE2EDuration="13.458597542s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.828853298s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:59.471609103 +0000 UTC m=+1185.955094838" watchObservedRunningTime="2026-09-17 22:18:06.458597542 +0000 UTC m=+1192.942083280"
Sep 17 22:18:10.185291 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:10.185265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:18:15.701753 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:15.701725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:18:15.704347 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:15.704321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:18:16.347219 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:16.347182    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"]
Sep 17 22:18:16.347443 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:16.347417    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" containerID="cri-o://84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71" gracePeriod=30
Sep 17 22:18:16.347523 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:16.347494    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kube-rbac-proxy" containerID="cri-o://b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b" gracePeriod=30
Sep 17 22:18:16.438951 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:16.438916    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:18:16.502634 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:16.502437    2563 generic.go:356] "Generic (PLEG): container finished" podID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerID="b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b" exitCode=2
Sep 17 22:18:16.502634 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:16.502558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerDied","Data":"b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b"}
Sep 17 22:18:20.180178 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:20.180127    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 22:18:20.184929 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:20.184903    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 22:18:21.090658 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.090635    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:18:21.275279 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls\") pod \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") "
Sep 17 22:18:21.275279 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jppxc\" (UniqueName: \"kubernetes.io/projected/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-api-access-jppxc\") pod \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") "
Sep 17 22:18:21.275752 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kserve-provision-location\") pod \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") "
Sep 17 22:18:21.275752 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-rbac-proxy-sar-config\") pod \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") "
Sep 17 22:18:21.275752 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-cabundle-cert\") pod \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\" (UID: \"38380e0d-f633-48d4-90f0-fcadd0f86fd0\") "
Sep 17 22:18:21.275752 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38380e0d-f633-48d4-90f0-fcadd0f86fd0" (UID: "38380e0d-f633-48d4-90f0-fcadd0f86fd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:18:21.275912 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38380e0d-f633-48d4-90f0-fcadd0f86fd0" (UID: "38380e0d-f633-48d4-90f0-fcadd0f86fd0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:18:21.275912 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.275831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38380e0d-f633-48d4-90f0-fcadd0f86fd0" (UID: "38380e0d-f633-48d4-90f0-fcadd0f86fd0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:18:21.277850 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.277828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38380e0d-f633-48d4-90f0-fcadd0f86fd0" (UID: "38380e0d-f633-48d4-90f0-fcadd0f86fd0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:18:21.277850 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.277833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-api-access-jppxc" (OuterVolumeSpecName: "kube-api-access-jppxc") pod "38380e0d-f633-48d4-90f0-fcadd0f86fd0" (UID: "38380e0d-f633-48d4-90f0-fcadd0f86fd0"). InnerVolumeSpecName "kube-api-access-jppxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:18:21.376558 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.376522    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:18:21.376558 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.376553    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:18:21.376746 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.376567    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38380e0d-f633-48d4-90f0-fcadd0f86fd0-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:18:21.376746 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.376581    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38380e0d-f633-48d4-90f0-fcadd0f86fd0-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:18:21.376746 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.376593    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jppxc\" (UniqueName: \"kubernetes.io/projected/38380e0d-f633-48d4-90f0-fcadd0f86fd0-kube-api-access-jppxc\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:18:21.524951 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.524916    2563 generic.go:356] "Generic (PLEG): container finished" podID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerID="84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71" exitCode=0
Sep 17 22:18:21.525123 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.524967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"
Sep 17 22:18:21.525123 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.525003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerDied","Data":"84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71"}
Sep 17 22:18:21.525123 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.525063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd" event={"ID":"38380e0d-f633-48d4-90f0-fcadd0f86fd0","Type":"ContainerDied","Data":"9b23d396eef379f66518ee09661a498576953eeb24d8baee5761bf423daedbab"}
Sep 17 22:18:21.525123 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.525089    2563 scope.go:117] "RemoveContainer" containerID="b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b"
Sep 17 22:18:21.533221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.533200    2563 scope.go:117] "RemoveContainer" containerID="84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71"
Sep 17 22:18:21.540139 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.540116    2563 scope.go:117] "RemoveContainer" containerID="54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7"
Sep 17 22:18:21.546203 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.546180    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"]
Sep 17 22:18:21.548028 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.548011    2563 scope.go:117] "RemoveContainer" containerID="b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b"
Sep 17 22:18:21.548412 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:18:21.548396    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b\": container with ID starting with b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b not found: ID does not exist" containerID="b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b"
Sep 17 22:18:21.548473 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.548419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b"} err="failed to get container status \"b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b\": rpc error: code = NotFound desc = could not find container \"b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b\": container with ID starting with b809a77fd58d2e1889fcacb59b3809974ef184299607e3e574c4a29f82cfd14b not found: ID does not exist"
Sep 17 22:18:21.548473 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.548436    2563 scope.go:117] "RemoveContainer" containerID="84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71"
Sep 17 22:18:21.548727 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:18:21.548707    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71\": container with ID starting with 84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71 not found: ID does not exist" containerID="84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71"
Sep 17 22:18:21.548797 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.548738    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71"} err="failed to get container status \"84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71\": rpc error: code = NotFound desc = could not find container \"84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71\": container with ID starting with 84da8f770ffa831b40d59107c5365fe543e354151c21c57f615e1ca5db63cd71 not found: ID does not exist"
Sep 17 22:18:21.548797 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.548761    2563 scope.go:117] "RemoveContainer" containerID="54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7"
Sep 17 22:18:21.549023 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:18:21.549008    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7\": container with ID starting with 54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7 not found: ID does not exist" containerID="54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7"
Sep 17 22:18:21.549064 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.549027    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7"} err="failed to get container status \"54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7\": rpc error: code = NotFound desc = could not find container \"54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7\": container with ID starting with 54c816d260d7685b682fd93536a9ead19e4530aca84d5e874e7e21608e3857b7 not found: ID does not exist"
Sep 17 22:18:21.549064 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.549042    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-eacee-predictor-6cd494fb8f-9rcjd"]
Sep 17 22:18:21.710207 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:21.710177    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" path="/var/lib/kubelet/pods/38380e0d-f633-48d4-90f0-fcadd0f86fd0/volumes"
Sep 17 22:18:26.439688 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:26.439648    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:18:36.439211 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:36.439169    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:18:46.439277 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:46.439230    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:18:56.439300 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:18:56.439262    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:19:06.439588 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:06.439506    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:19:15.716126 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:15.716080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:19:23.154345 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:23.154310    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"]
Sep 17 22:19:23.154747 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:23.154626    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" containerID="cri-o://9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81" gracePeriod=30
Sep 17 22:19:23.154747 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:23.154646    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kube-rbac-proxy" containerID="cri-o://78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1" gracePeriod=30
Sep 17 22:19:23.771511 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:23.771478    2563 generic.go:356] "Generic (PLEG): container finished" podID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerID="78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1" exitCode=2
Sep 17 22:19:23.771672 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:23.771531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerDied","Data":"78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1"}
Sep 17 22:19:25.711456 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:25.711417    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 22:19:26.435276 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:26.435231    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 22:19:27.497726 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.497701    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:19:27.501920 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.501902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-cabundle-cert\") pod \"f123cc3a-f638-4be8-a686-69c4dacd8715\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") "
Sep 17 22:19:27.502002 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.501936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f123cc3a-f638-4be8-a686-69c4dacd8715-kserve-provision-location\") pod \"f123cc3a-f638-4be8-a686-69c4dacd8715\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") "
Sep 17 22:19:27.502002 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.501976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f123cc3a-f638-4be8-a686-69c4dacd8715-proxy-tls\") pod \"f123cc3a-f638-4be8-a686-69c4dacd8715\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") "
Sep 17 22:19:27.502002 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.502000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-rbac-proxy-sar-config\") pod \"f123cc3a-f638-4be8-a686-69c4dacd8715\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") "
Sep 17 22:19:27.502188 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.502017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flhtz\" (UniqueName: \"kubernetes.io/projected/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-api-access-flhtz\") pod \"f123cc3a-f638-4be8-a686-69c4dacd8715\" (UID: \"f123cc3a-f638-4be8-a686-69c4dacd8715\") "
Sep 17 22:19:27.502330 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.502310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f123cc3a-f638-4be8-a686-69c4dacd8715-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f123cc3a-f638-4be8-a686-69c4dacd8715" (UID: "f123cc3a-f638-4be8-a686-69c4dacd8715"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:27.502383 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.502328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f123cc3a-f638-4be8-a686-69c4dacd8715" (UID: "f123cc3a-f638-4be8-a686-69c4dacd8715"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:27.502383 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.502364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f123cc3a-f638-4be8-a686-69c4dacd8715" (UID: "f123cc3a-f638-4be8-a686-69c4dacd8715"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:27.503998 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.503979    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f123cc3a-f638-4be8-a686-69c4dacd8715-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f123cc3a-f638-4be8-a686-69c4dacd8715" (UID: "f123cc3a-f638-4be8-a686-69c4dacd8715"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:19:27.504375 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.504349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-api-access-flhtz" (OuterVolumeSpecName: "kube-api-access-flhtz") pod "f123cc3a-f638-4be8-a686-69c4dacd8715" (UID: "f123cc3a-f638-4be8-a686-69c4dacd8715"). InnerVolumeSpecName "kube-api-access-flhtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:27.602560 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.602534    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f123cc3a-f638-4be8-a686-69c4dacd8715-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.602560 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.602558    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.602713 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.602568    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flhtz\" (UniqueName: \"kubernetes.io/projected/f123cc3a-f638-4be8-a686-69c4dacd8715-kube-api-access-flhtz\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.602713 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.602577    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f123cc3a-f638-4be8-a686-69c4dacd8715-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.602713 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.602586    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f123cc3a-f638-4be8-a686-69c4dacd8715-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.788266 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.788237    2563 generic.go:356] "Generic (PLEG): container finished" podID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerID="9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81" exitCode=0
Sep 17 22:19:27.788405 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.788291    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"
Sep 17 22:19:27.788405 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.788314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerDied","Data":"9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81"}
Sep 17 22:19:27.788405 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.788373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn" event={"ID":"f123cc3a-f638-4be8-a686-69c4dacd8715","Type":"ContainerDied","Data":"43ab2671101949c2734b87be0ec07241f60d59fd05a5bd0b2a9d57c58308b704"}
Sep 17 22:19:27.788405 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.788400    2563 scope.go:117] "RemoveContainer" containerID="78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1"
Sep 17 22:19:27.796271 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.796256    2563 scope.go:117] "RemoveContainer" containerID="9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81"
Sep 17 22:19:27.803042 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.803005    2563 scope.go:117] "RemoveContainer" containerID="c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1"
Sep 17 22:19:27.805363 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.805340    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"]
Sep 17 22:19:27.809024 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.809004    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-4vtvn"]
Sep 17 22:19:27.810471 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.810454    2563 scope.go:117] "RemoveContainer" containerID="78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1"
Sep 17 22:19:27.810726 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:19:27.810706    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1\": container with ID starting with 78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1 not found: ID does not exist" containerID="78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1"
Sep 17 22:19:27.810793 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.810738    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1"} err="failed to get container status \"78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1\": rpc error: code = NotFound desc = could not find container \"78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1\": container with ID starting with 78d3cb5d7f6db1ffa721affdb6da3ea7297577caf87678ebc36b84eece1068a1 not found: ID does not exist"
Sep 17 22:19:27.810793 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.810761    2563 scope.go:117] "RemoveContainer" containerID="9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81"
Sep 17 22:19:27.811021 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:19:27.811004    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81\": container with ID starting with 9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81 not found: ID does not exist" containerID="9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81"
Sep 17 22:19:27.811061 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.811026    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81"} err="failed to get container status \"9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81\": rpc error: code = NotFound desc = could not find container \"9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81\": container with ID starting with 9c5e9c1d95f4a596d7d6da79e78a21b7bd8dda195e56c71f63aaa5098fe61f81 not found: ID does not exist"
Sep 17 22:19:27.811061 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.811040    2563 scope.go:117] "RemoveContainer" containerID="c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1"
Sep 17 22:19:27.811275 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:19:27.811258    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1\": container with ID starting with c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1 not found: ID does not exist" containerID="c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1"
Sep 17 22:19:27.811333 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:27.811277    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1"} err="failed to get container status \"c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1\": rpc error: code = NotFound desc = could not find container \"c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1\": container with ID starting with c29a44a987c12d4d0b02513adcb437a030a3a594460e77677c93cf12aea23bd1 not found: ID does not exist"
Sep 17 22:19:28.369491 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"]
Sep 17 22:19:28.369744 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kube-rbac-proxy"
Sep 17 22:19:28.369790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369745    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kube-rbac-proxy"
Sep 17 22:19:28.369790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369757    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container"
Sep 17 22:19:28.369790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369764    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container"
Sep 17 22:19:28.369790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369776    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="storage-initializer"
Sep 17 22:19:28.369790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369782    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="storage-initializer"
Sep 17 22:19:28.369790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369790    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="storage-initializer"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369796    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="storage-initializer"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369803    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kube-rbac-proxy"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369808    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kube-rbac-proxy"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369818    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369866    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kube-rbac-proxy"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369874    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38380e0d-f633-48d4-90f0-fcadd0f86fd0" containerName="kserve-container"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369879    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kserve-container"
Sep 17 22:19:28.370257 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.369884    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" containerName="kube-rbac-proxy"
Sep 17 22:19:28.374436 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.374420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.377156 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.377097    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:19:28.377156 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.377130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 22:19:28.377317 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.377162    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:19:28.377317 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.377181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 22:19:28.377317 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.377227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:19:28.377511 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.377496    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:19:28.377558 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.377538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:19:28.382281 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.382261    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"]
Sep 17 22:19:28.408418 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.408395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.408514 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.408425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.408514 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.408468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5b194d9-b982-4802-8165-3f4a31e05525-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.408514 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.408501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5b194d9-b982-4802-8165-3f4a31e05525-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.408620 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.408526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc7mm\" (UniqueName: \"kubernetes.io/projected/e5b194d9-b982-4802-8165-3f4a31e05525-kube-api-access-mc7mm\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509164 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc7mm\" (UniqueName: \"kubernetes.io/projected/e5b194d9-b982-4802-8165-3f4a31e05525-kube-api-access-mc7mm\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509454 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509454 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509454 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5b194d9-b982-4802-8165-3f4a31e05525-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509454 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5b194d9-b982-4802-8165-3f4a31e05525-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509738 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5b194d9-b982-4802-8165-3f4a31e05525-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509955 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.509994 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.509945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.511764 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.511746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5b194d9-b982-4802-8165-3f4a31e05525-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.516694 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.516663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc7mm\" (UniqueName: \"kubernetes.io/projected/e5b194d9-b982-4802-8165-3f4a31e05525-kube-api-access-mc7mm\") pod \"isvc-canary-promote-predictor-567686cc76-sss9k\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.685833 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.685763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:28.958661 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:28.958583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"]
Sep 17 22:19:28.961230 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:19:28.961202    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b194d9_b982_4802_8165_3f4a31e05525.slice/crio-b046a26c16486c31bf79047e60ace81339cdaa087e4ba90a8677a2a0fde9d6d3 WatchSource:0}: Error finding container b046a26c16486c31bf79047e60ace81339cdaa087e4ba90a8677a2a0fde9d6d3: Status 404 returned error can't find the container with id b046a26c16486c31bf79047e60ace81339cdaa087e4ba90a8677a2a0fde9d6d3
Sep 17 22:19:29.710381 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:29.710351    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f123cc3a-f638-4be8-a686-69c4dacd8715" path="/var/lib/kubelet/pods/f123cc3a-f638-4be8-a686-69c4dacd8715/volumes"
Sep 17 22:19:29.798799 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:29.798761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerStarted","Data":"b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae"}
Sep 17 22:19:29.798969 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:29.798832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerStarted","Data":"b046a26c16486c31bf79047e60ace81339cdaa087e4ba90a8677a2a0fde9d6d3"}
Sep 17 22:19:33.817790 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:33.817755    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5b194d9-b982-4802-8165-3f4a31e05525" containerID="b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae" exitCode=0
Sep 17 22:19:33.818284 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:33.817827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerDied","Data":"b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae"}
Sep 17 22:19:34.824837 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:34.824802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerStarted","Data":"0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9"}
Sep 17 22:19:34.825233 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:34.824867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerStarted","Data":"53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136"}
Sep 17 22:19:34.825233 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:34.825060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:34.843551 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:34.843178    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podStartSLOduration=2.861576095 podStartE2EDuration="6.843160346s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.981584251s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:19:34.8418613 +0000 UTC m=+1281.325347034" watchObservedRunningTime="2026-09-17 22:19:34.843160346 +0000 UTC m=+1281.326646082"
Sep 17 22:19:35.829370 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:35.829333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:35.830470 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:35.830443    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:19:36.832589 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:36.832547    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:19:41.837069 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:41.837035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:19:41.837608 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:41.837577    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:19:51.837915 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:19:51.837880    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:20:01.837480 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:01.837443    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:20:11.837691 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:11.837649    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:20:21.838300 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:21.838254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:20:31.837565 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:31.837471    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:20:41.838530 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:41.838489    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 22:20:51.838944 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:51.838917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:20:53.581457 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:53.581405    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"]
Sep 17 22:20:53.581861 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:53.581651    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container" containerID="cri-o://53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136" gracePeriod=30
Sep 17 22:20:53.581861 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:53.581725    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kube-rbac-proxy" containerID="cri-o://0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9" gracePeriod=30
Sep 17 22:20:54.143054 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:54.143024    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5b194d9-b982-4802-8165-3f4a31e05525" containerID="0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9" exitCode=2
Sep 17 22:20:54.143234 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:54.143098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerDied","Data":"0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9"}
Sep 17 22:20:56.832957 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:56.832911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 22:20:58.507329 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.507297    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"]
Sep 17 22:20:58.510824 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.510807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.512926 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.512907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 22:20:58.520825 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.520797    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"]
Sep 17 22:20:58.530290 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.530264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:20:58.594522 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5b194d9-b982-4802-8165-3f4a31e05525-proxy-tls\") pod \"e5b194d9-b982-4802-8165-3f4a31e05525\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") "
Sep 17 22:20:58.594729 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594530    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5b194d9-b982-4802-8165-3f4a31e05525-kserve-provision-location\") pod \"e5b194d9-b982-4802-8165-3f4a31e05525\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") "
Sep 17 22:20:58.594729 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc7mm\" (UniqueName: \"kubernetes.io/projected/e5b194d9-b982-4802-8165-3f4a31e05525-kube-api-access-mc7mm\") pod \"e5b194d9-b982-4802-8165-3f4a31e05525\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") "
Sep 17 22:20:58.594729 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594612    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-kube-rbac-proxy-sar-config\") pod \"e5b194d9-b982-4802-8165-3f4a31e05525\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") "
Sep 17 22:20:58.594729 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-cabundle-cert\") pod \"e5b194d9-b982-4802-8165-3f4a31e05525\" (UID: \"e5b194d9-b982-4802-8165-3f4a31e05525\") "
Sep 17 22:20:58.594949 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/314ad144-d635-4bdf-839a-c2c660305399-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.594949 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.594949 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.594949 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b194d9-b982-4802-8165-3f4a31e05525-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5b194d9-b982-4802-8165-3f4a31e05525" (UID: "e5b194d9-b982-4802-8165-3f4a31e05525"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:20:58.594949 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289xx\" (UniqueName: \"kubernetes.io/projected/314ad144-d635-4bdf-839a-c2c660305399-kube-api-access-289xx\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.595227 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.594963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5b194d9-b982-4802-8165-3f4a31e05525" (UID: "e5b194d9-b982-4802-8165-3f4a31e05525"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:20:58.595227 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.595066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5b194d9-b982-4802-8165-3f4a31e05525" (UID: "e5b194d9-b982-4802-8165-3f4a31e05525"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:20:58.595227 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.595125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/314ad144-d635-4bdf-839a-c2c660305399-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.595227 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.595205    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:20:58.595227 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.595222    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5b194d9-b982-4802-8165-3f4a31e05525-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:20:58.595489 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.595239    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5b194d9-b982-4802-8165-3f4a31e05525-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:20:58.596747 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.596726    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b194d9-b982-4802-8165-3f4a31e05525-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5b194d9-b982-4802-8165-3f4a31e05525" (UID: "e5b194d9-b982-4802-8165-3f4a31e05525"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:20:58.596854 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.596832    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b194d9-b982-4802-8165-3f4a31e05525-kube-api-access-mc7mm" (OuterVolumeSpecName: "kube-api-access-mc7mm") pod "e5b194d9-b982-4802-8165-3f4a31e05525" (UID: "e5b194d9-b982-4802-8165-3f4a31e05525"). InnerVolumeSpecName "kube-api-access-mc7mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:20:58.696356 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/314ad144-d635-4bdf-839a-c2c660305399-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.696356 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.696536 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.696536 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289xx\" (UniqueName: \"kubernetes.io/projected/314ad144-d635-4bdf-839a-c2c660305399-kube-api-access-289xx\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.696536 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/314ad144-d635-4bdf-839a-c2c660305399-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.696657 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696609    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5b194d9-b982-4802-8165-3f4a31e05525-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:20:58.696657 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696629    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc7mm\" (UniqueName: \"kubernetes.io/projected/e5b194d9-b982-4802-8165-3f4a31e05525-kube-api-access-mc7mm\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:20:58.696732 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/314ad144-d635-4bdf-839a-c2c660305399-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.697003 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.696983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.697061 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.697045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.698929 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.698911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/314ad144-d635-4bdf-839a-c2c660305399-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.704234 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.704210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-289xx\" (UniqueName: \"kubernetes.io/projected/314ad144-d635-4bdf-839a-c2c660305399-kube-api-access-289xx\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:58.835289 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:58.835249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:20:59.107799 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.107766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"]
Sep 17 22:20:59.112848 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:20:59.112822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314ad144_d635_4bdf_839a_c2c660305399.slice/crio-19dcd2df5ac4483e32ac78617612abe16a694ef6a13853d8d0e131fb1e7735b9 WatchSource:0}: Error finding container 19dcd2df5ac4483e32ac78617612abe16a694ef6a13853d8d0e131fb1e7735b9: Status 404 returned error can't find the container with id 19dcd2df5ac4483e32ac78617612abe16a694ef6a13853d8d0e131fb1e7735b9
Sep 17 22:20:59.165272 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.165245    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5b194d9-b982-4802-8165-3f4a31e05525" containerID="53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136" exitCode=0
Sep 17 22:20:59.165404 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.165303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"
Sep 17 22:20:59.165404 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.165330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerDied","Data":"53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136"}
Sep 17 22:20:59.165523 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.165411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k" event={"ID":"e5b194d9-b982-4802-8165-3f4a31e05525","Type":"ContainerDied","Data":"b046a26c16486c31bf79047e60ace81339cdaa087e4ba90a8677a2a0fde9d6d3"}
Sep 17 22:20:59.165523 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.165447    2563 scope.go:117] "RemoveContainer" containerID="0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9"
Sep 17 22:20:59.167156 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.167128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerStarted","Data":"19dcd2df5ac4483e32ac78617612abe16a694ef6a13853d8d0e131fb1e7735b9"}
Sep 17 22:20:59.174405 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.174317    2563 scope.go:117] "RemoveContainer" containerID="53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136"
Sep 17 22:20:59.181805 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.181789    2563 scope.go:117] "RemoveContainer" containerID="b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae"
Sep 17 22:20:59.187378 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.187356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"]
Sep 17 22:20:59.191078 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.191053    2563 scope.go:117] "RemoveContainer" containerID="0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9"
Sep 17 22:20:59.191450 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:20:59.191423    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9\": container with ID starting with 0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9 not found: ID does not exist" containerID="0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9"
Sep 17 22:20:59.191524 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.191460    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9"} err="failed to get container status \"0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9\": rpc error: code = NotFound desc = could not find container \"0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9\": container with ID starting with 0108e79cd317f3b8997fc1672f85e71d87610c9f12eae4b2f74fcdd3f837c2a9 not found: ID does not exist"
Sep 17 22:20:59.191524 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.191479    2563 scope.go:117] "RemoveContainer" containerID="53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136"
Sep 17 22:20:59.191767 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:20:59.191750    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136\": container with ID starting with 53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136 not found: ID does not exist" containerID="53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136"
Sep 17 22:20:59.191820 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.191771    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136"} err="failed to get container status \"53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136\": rpc error: code = NotFound desc = could not find container \"53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136\": container with ID starting with 53e4c1d999e86151034afb146d86c69dc91a1fb00b2c8be1217d81be04419136 not found: ID does not exist"
Sep 17 22:20:59.191820 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.191785    2563 scope.go:117] "RemoveContainer" containerID="b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae"
Sep 17 22:20:59.192040 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:20:59.192025    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae\": container with ID starting with b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae not found: ID does not exist" containerID="b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae"
Sep 17 22:20:59.192087 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.192044    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae"} err="failed to get container status \"b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae\": rpc error: code = NotFound desc = could not find container \"b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae\": container with ID starting with b441c7da5e41e82ba1d7a7b5f68cd62b7c326eab3841754d2e421690be9334ae not found: ID does not exist"
Sep 17 22:20:59.192208 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.192190    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-sss9k"]
Sep 17 22:20:59.710091 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:20:59.710055    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" path="/var/lib/kubelet/pods/e5b194d9-b982-4802-8165-3f4a31e05525/volumes"
Sep 17 22:21:00.173117 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:00.173016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerStarted","Data":"61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612"}
Sep 17 22:21:03.186506 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:03.186467    2563 generic.go:356] "Generic (PLEG): container finished" podID="314ad144-d635-4bdf-839a-c2c660305399" containerID="61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612" exitCode=0
Sep 17 22:21:03.186887 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:03.186528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerDied","Data":"61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612"}
Sep 17 22:21:04.192430 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:04.192399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerStarted","Data":"59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3"}
Sep 17 22:21:04.192840 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:04.192466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerStarted","Data":"ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9"}
Sep 17 22:21:04.192840 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:04.192672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:21:04.211603 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:04.211546    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podStartSLOduration=2.38741518 podStartE2EDuration="6.211528792s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824113612s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:21:04.20962521 +0000 UTC m=+1370.693110955" watchObservedRunningTime="2026-09-17 22:21:04.211528792 +0000 UTC m=+1370.695014702"
Sep 17 22:21:05.197064 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:05.197031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:21:05.198189 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:05.198164    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:21:06.200976 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:06.200938    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:21:11.207027 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:11.206998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:21:11.207580 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:11.207551    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:21:21.207731 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:21.207691    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:21:31.208265 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:31.208227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:21:41.208201 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:41.208162    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:21:51.207686 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:21:51.207638    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:22:01.207959 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:01.207880    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:22:11.208016 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:11.207978    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:22:21.209048 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:21.209020    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:22:28.675165 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:28.675133    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"]
Sep 17 22:22:28.675667 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:28.675386    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" containerID="cri-o://ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9" gracePeriod=30
Sep 17 22:22:28.675667 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:28.675426    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kube-rbac-proxy" containerID="cri-o://59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3" gracePeriod=30
Sep 17 22:22:29.517013 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:29.516978    2563 generic.go:356] "Generic (PLEG): container finished" podID="314ad144-d635-4bdf-839a-c2c660305399" containerID="59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3" exitCode=2
Sep 17 22:22:29.517204 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:29.517041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerDied","Data":"59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3"}
Sep 17 22:22:31.201770 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:31.201727    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 22:22:31.208181 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:31.208147    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 22:22:33.315438 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.315415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:22:33.462542 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.462514    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-kube-rbac-proxy-sar-config\") pod \"314ad144-d635-4bdf-839a-c2c660305399\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") "
Sep 17 22:22:33.462705 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.462608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-289xx\" (UniqueName: \"kubernetes.io/projected/314ad144-d635-4bdf-839a-c2c660305399-kube-api-access-289xx\") pod \"314ad144-d635-4bdf-839a-c2c660305399\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") "
Sep 17 22:22:33.462705 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.462640    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/314ad144-d635-4bdf-839a-c2c660305399-kserve-provision-location\") pod \"314ad144-d635-4bdf-839a-c2c660305399\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") "
Sep 17 22:22:33.462705 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.462669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/314ad144-d635-4bdf-839a-c2c660305399-proxy-tls\") pod \"314ad144-d635-4bdf-839a-c2c660305399\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") "
Sep 17 22:22:33.462705 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.462687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-cabundle-cert\") pod \"314ad144-d635-4bdf-839a-c2c660305399\" (UID: \"314ad144-d635-4bdf-839a-c2c660305399\") "
Sep 17 22:22:33.462984 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.462938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "314ad144-d635-4bdf-839a-c2c660305399" (UID: "314ad144-d635-4bdf-839a-c2c660305399"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:22:33.463059 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.462972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314ad144-d635-4bdf-839a-c2c660305399-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "314ad144-d635-4bdf-839a-c2c660305399" (UID: "314ad144-d635-4bdf-839a-c2c660305399"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:22:33.463204 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.463179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "314ad144-d635-4bdf-839a-c2c660305399" (UID: "314ad144-d635-4bdf-839a-c2c660305399"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:22:33.464755 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.464729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314ad144-d635-4bdf-839a-c2c660305399-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "314ad144-d635-4bdf-839a-c2c660305399" (UID: "314ad144-d635-4bdf-839a-c2c660305399"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:22:33.464810 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.464788    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314ad144-d635-4bdf-839a-c2c660305399-kube-api-access-289xx" (OuterVolumeSpecName: "kube-api-access-289xx") pod "314ad144-d635-4bdf-839a-c2c660305399" (UID: "314ad144-d635-4bdf-839a-c2c660305399"). InnerVolumeSpecName "kube-api-access-289xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:22:33.533360 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.533325    2563 generic.go:356] "Generic (PLEG): container finished" podID="314ad144-d635-4bdf-839a-c2c660305399" containerID="ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9" exitCode=0
Sep 17 22:22:33.533493 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.533386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"
Sep 17 22:22:33.533493 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.533399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerDied","Data":"ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9"}
Sep 17 22:22:33.533493 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.533457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl" event={"ID":"314ad144-d635-4bdf-839a-c2c660305399","Type":"ContainerDied","Data":"19dcd2df5ac4483e32ac78617612abe16a694ef6a13853d8d0e131fb1e7735b9"}
Sep 17 22:22:33.533493 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.533483    2563 scope.go:117] "RemoveContainer" containerID="59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3"
Sep 17 22:22:33.542544 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.542526    2563 scope.go:117] "RemoveContainer" containerID="ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9"
Sep 17 22:22:33.549507 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.549489    2563 scope.go:117] "RemoveContainer" containerID="61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612"
Sep 17 22:22:33.555379 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.555356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"]
Sep 17 22:22:33.556531 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.556518    2563 scope.go:117] "RemoveContainer" containerID="59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3"
Sep 17 22:22:33.556777 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:22:33.556762    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3\": container with ID starting with 59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3 not found: ID does not exist" containerID="59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3"
Sep 17 22:22:33.556827 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.556783    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3"} err="failed to get container status \"59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3\": rpc error: code = NotFound desc = could not find container \"59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3\": container with ID starting with 59ba02d1b1c09d49de07d0274e56603004e5b336fc22db9edff1d058422332d3 not found: ID does not exist"
Sep 17 22:22:33.556827 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.556798    2563 scope.go:117] "RemoveContainer" containerID="ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9"
Sep 17 22:22:33.557045 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:22:33.557023    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9\": container with ID starting with ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9 not found: ID does not exist" containerID="ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9"
Sep 17 22:22:33.557144 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.557047    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9"} err="failed to get container status \"ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9\": rpc error: code = NotFound desc = could not find container \"ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9\": container with ID starting with ccb81d7860c610a2bfa4f9a33c97d89219dfc669fa9af49a61f5a2a13d1f82d9 not found: ID does not exist"
Sep 17 22:22:33.557144 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.557060    2563 scope.go:117] "RemoveContainer" containerID="61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612"
Sep 17 22:22:33.557477 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:22:33.557400    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612\": container with ID starting with 61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612 not found: ID does not exist" containerID="61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612"
Sep 17 22:22:33.557477 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.557435    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612"} err="failed to get container status \"61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612\": rpc error: code = NotFound desc = could not find container \"61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612\": container with ID starting with 61651fa7b8cf5e837558d0636fc8b2048609618a55dd0142cdc5459c89f45612 not found: ID does not exist"
Sep 17 22:22:33.558997 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.558979    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-d4zjl"]
Sep 17 22:22:33.563385 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.563368    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:22:33.563449 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.563387    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289xx\" (UniqueName: \"kubernetes.io/projected/314ad144-d635-4bdf-839a-c2c660305399-kube-api-access-289xx\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:22:33.563449 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.563398    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/314ad144-d635-4bdf-839a-c2c660305399-kserve-provision-location\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:22:33.563449 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.563407    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/314ad144-d635-4bdf-839a-c2c660305399-proxy-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:22:33.563449 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.563417    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/314ad144-d635-4bdf-839a-c2c660305399-cabundle-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:22:33.709736 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:22:33.709707    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="314ad144-d635-4bdf-839a-c2c660305399" path="/var/lib/kubelet/pods/314ad144-d635-4bdf-839a-c2c660305399/volumes"
Sep 17 22:23:15.728663 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:23:15.728638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:23:15.732522 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:23:15.732498    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:24:28.706256 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706224    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jrksc/must-gather-np8bt"]
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706516    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706527    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="storage-initializer"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="storage-initializer"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706559    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706571    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kube-rbac-proxy"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706577    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kube-rbac-proxy"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706586    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kube-rbac-proxy"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kube-rbac-proxy"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="storage-initializer"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706604    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="storage-initializer"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706646    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kube-rbac-proxy"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706654    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b194d9-b982-4802-8165-3f4a31e05525" containerName="kserve-container"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706660    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kube-rbac-proxy"
Sep 17 22:24:28.708673 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.706665    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="314ad144-d635-4bdf-839a-c2c660305399" containerName="kserve-container"
Sep 17 22:24:28.709482 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.709466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:28.712074 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.712054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jrksc\"/\"kube-root-ca.crt\""
Sep 17 22:24:28.712976 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.712955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jrksc\"/\"openshift-service-ca.crt\""
Sep 17 22:24:28.713083 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.712955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jrksc\"/\"default-dockercfg-9kgdk\""
Sep 17 22:24:28.718546 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.718518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jrksc/must-gather-np8bt"]
Sep 17 22:24:28.834013 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.833976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8b79c7c-d673-45f2-b0a0-432a572090b7-must-gather-output\") pod \"must-gather-np8bt\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") " pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:28.834184 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.834023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkz8f\" (UniqueName: \"kubernetes.io/projected/d8b79c7c-d673-45f2-b0a0-432a572090b7-kube-api-access-kkz8f\") pod \"must-gather-np8bt\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") " pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:28.934998 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.934969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8b79c7c-d673-45f2-b0a0-432a572090b7-must-gather-output\") pod \"must-gather-np8bt\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") " pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:28.935160 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.935008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkz8f\" (UniqueName: \"kubernetes.io/projected/d8b79c7c-d673-45f2-b0a0-432a572090b7-kube-api-access-kkz8f\") pod \"must-gather-np8bt\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") " pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:28.935328 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.935303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8b79c7c-d673-45f2-b0a0-432a572090b7-must-gather-output\") pod \"must-gather-np8bt\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") " pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:28.943386 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:28.943363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkz8f\" (UniqueName: \"kubernetes.io/projected/d8b79c7c-d673-45f2-b0a0-432a572090b7-kube-api-access-kkz8f\") pod \"must-gather-np8bt\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") " pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:29.019755 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:29.019689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:24:29.309846 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:29.309770    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jrksc/must-gather-np8bt"]
Sep 17 22:24:29.313544 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:24:29.313518    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8b79c7c_d673_45f2_b0a0_432a572090b7.slice/crio-e83dddb4d244685076f5a7195bf3eb49e1ebfc1d86097c12740bcaa453b14f4b WatchSource:0}: Error finding container e83dddb4d244685076f5a7195bf3eb49e1ebfc1d86097c12740bcaa453b14f4b: Status 404 returned error can't find the container with id e83dddb4d244685076f5a7195bf3eb49e1ebfc1d86097c12740bcaa453b14f4b
Sep 17 22:24:29.315246 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:29.315227    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:24:29.978116 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:29.978070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jrksc/must-gather-np8bt" event={"ID":"d8b79c7c-d673-45f2-b0a0-432a572090b7","Type":"ContainerStarted","Data":"e83dddb4d244685076f5a7195bf3eb49e1ebfc1d86097c12740bcaa453b14f4b"}
Sep 17 22:24:36.007246 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:36.007210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jrksc/must-gather-np8bt" event={"ID":"d8b79c7c-d673-45f2-b0a0-432a572090b7","Type":"ContainerStarted","Data":"eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05"}
Sep 17 22:24:36.007246 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:36.007254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jrksc/must-gather-np8bt" event={"ID":"d8b79c7c-d673-45f2-b0a0-432a572090b7","Type":"ContainerStarted","Data":"37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575"}
Sep 17 22:24:36.023169 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:36.023120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jrksc/must-gather-np8bt" podStartSLOduration=1.885434113 podStartE2EDuration="8.023091084s" totalImagesPullingTime="6.137656971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:36.02196936 +0000 UTC m=+1582.505455110" watchObservedRunningTime="2026-09-17 22:24:36.023091084 +0000 UTC m=+1582.506576817"
Sep 17 22:24:59.123068 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:59.123032    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerID="37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575" exitCode=0
Sep 17 22:24:59.123629 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:59.123119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jrksc/must-gather-np8bt" event={"ID":"d8b79c7c-d673-45f2-b0a0-432a572090b7","Type":"ContainerDied","Data":"37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575"}
Sep 17 22:24:59.123629 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:59.123475    2563 scope.go:117] "RemoveContainer" containerID="37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575"
Sep 17 22:24:59.291658 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:24:59.291623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jrksc_must-gather-np8bt_d8b79c7c-d673-45f2-b0a0-432a572090b7/gather/0.log"
Sep 17 22:25:02.718965 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:02.718938    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sgwj2_24916e2a-7f9f-4375-a023-033d9af6d468/global-pull-secret-syncer/0.log"
Sep 17 22:25:02.833773 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:02.833744    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n8vq4_2892cc96-3ca8-42db-b1cb-363cac3b86de/konnectivity-agent/0.log"
Sep 17 22:25:02.940957 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:02.940922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-88.ec2.internal_5cce065d5c7d1b9816d8d702b1174775/haproxy/0.log"
Sep 17 22:25:04.707423 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:04.707385    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jrksc/must-gather-np8bt"]
Sep 17 22:25:04.707860 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:04.707558    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jrksc/must-gather-np8bt" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerName="copy" containerID="cri-o://eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05" gracePeriod=2
Sep 17 22:25:04.711479 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:04.711088    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jrksc/must-gather-np8bt"]
Sep 17 22:25:04.933081 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:04.933059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jrksc_must-gather-np8bt_d8b79c7c-d673-45f2-b0a0-432a572090b7/copy/0.log"
Sep 17 22:25:04.934574 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:04.934557    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:25:04.936627 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:25:04.936603    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-np8bt\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jrksc\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:25:05.039351 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.039255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkz8f\" (UniqueName: \"kubernetes.io/projected/d8b79c7c-d673-45f2-b0a0-432a572090b7-kube-api-access-kkz8f\") pod \"d8b79c7c-d673-45f2-b0a0-432a572090b7\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") "
Sep 17 22:25:05.039351 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.039303    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8b79c7c-d673-45f2-b0a0-432a572090b7-must-gather-output\") pod \"d8b79c7c-d673-45f2-b0a0-432a572090b7\" (UID: \"d8b79c7c-d673-45f2-b0a0-432a572090b7\") "
Sep 17 22:25:05.041404 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.041379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b79c7c-d673-45f2-b0a0-432a572090b7-kube-api-access-kkz8f" (OuterVolumeSpecName: "kube-api-access-kkz8f") pod "d8b79c7c-d673-45f2-b0a0-432a572090b7" (UID: "d8b79c7c-d673-45f2-b0a0-432a572090b7"). InnerVolumeSpecName "kube-api-access-kkz8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:25:05.041517 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.041379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b79c7c-d673-45f2-b0a0-432a572090b7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d8b79c7c-d673-45f2-b0a0-432a572090b7" (UID: "d8b79c7c-d673-45f2-b0a0-432a572090b7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:05.140844 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.140809    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkz8f\" (UniqueName: \"kubernetes.io/projected/d8b79c7c-d673-45f2-b0a0-432a572090b7-kube-api-access-kkz8f\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:25:05.140844 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.140839    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8b79c7c-d673-45f2-b0a0-432a572090b7-must-gather-output\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 17 22:25:05.148009 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.147988    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jrksc_must-gather-np8bt_d8b79c7c-d673-45f2-b0a0-432a572090b7/copy/0.log"
Sep 17 22:25:05.148330 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.148310    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerID="eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05" exitCode=143
Sep 17 22:25:05.148418 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.148367    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:25:05.148418 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.148407    2563 scope.go:117] "RemoveContainer" containerID="eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05"
Sep 17 22:25:05.150490 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:25:05.150464    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-np8bt\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jrksc\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:25:05.156690 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.156668    2563 scope.go:117] "RemoveContainer" containerID="37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575"
Sep 17 22:25:05.159294 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:25:05.159265    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-np8bt\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jrksc\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:25:05.169098 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.169082    2563 scope.go:117] "RemoveContainer" containerID="eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05"
Sep 17 22:25:05.169396 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:25:05.169375    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05\": container with ID starting with eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05 not found: ID does not exist" containerID="eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05"
Sep 17 22:25:05.169473 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.169409    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05"} err="failed to get container status \"eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05\": rpc error: code = NotFound desc = could not find container \"eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05\": container with ID starting with eb638688ec9fa6acaaae521b924e8bc525f41e735df353e718b7cef94c4dae05 not found: ID does not exist"
Sep 17 22:25:05.169473 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.169450    2563 scope.go:117] "RemoveContainer" containerID="37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575"
Sep 17 22:25:05.169688 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:25:05.169672    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575\": container with ID starting with 37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575 not found: ID does not exist" containerID="37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575"
Sep 17 22:25:05.169734 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.169694    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575"} err="failed to get container status \"37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575\": rpc error: code = NotFound desc = could not find container \"37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575\": container with ID starting with 37ed087d3c3a850ba6353e9535ee93c947f3269fcc938ce340ca3774bc03e575 not found: ID does not exist"
Sep 17 22:25:05.709712 ip-10-0-139-88 kubenswrapper[2563]: E0917 22:25:05.709674    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-np8bt\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jrksc\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" pod="openshift-must-gather-jrksc/must-gather-np8bt"
Sep 17 22:25:05.710590 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:05.710568    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" path="/var/lib/kubelet/pods/d8b79c7c-d673-45f2-b0a0-432a572090b7/volumes"
Sep 17 22:25:07.832518 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:07.832488    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rxq5z_ef457f63-4426-4082-9ee8-398a6210329e/node-exporter/0.log"
Sep 17 22:25:07.853938 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:07.853914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rxq5z_ef457f63-4426-4082-9ee8-398a6210329e/kube-rbac-proxy/0.log"
Sep 17 22:25:07.880361 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:07.880340    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rxq5z_ef457f63-4426-4082-9ee8-398a6210329e/init-textfile/0.log"
Sep 17 22:25:08.091492 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.091460    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ed5a58f-f659-493f-a5fc-ecfd1b514221/prometheus/0.log"
Sep 17 22:25:08.132497 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.132459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ed5a58f-f659-493f-a5fc-ecfd1b514221/config-reloader/0.log"
Sep 17 22:25:08.168290 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.168260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ed5a58f-f659-493f-a5fc-ecfd1b514221/thanos-sidecar/0.log"
Sep 17 22:25:08.214965 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.214943    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ed5a58f-f659-493f-a5fc-ecfd1b514221/kube-rbac-proxy-web/0.log"
Sep 17 22:25:08.251174 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.251135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ed5a58f-f659-493f-a5fc-ecfd1b514221/kube-rbac-proxy/0.log"
Sep 17 22:25:08.283808 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.283768    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ed5a58f-f659-493f-a5fc-ecfd1b514221/kube-rbac-proxy-thanos/0.log"
Sep 17 22:25:08.310821 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.310795    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ed5a58f-f659-493f-a5fc-ecfd1b514221/init-config-reloader/0.log"
Sep 17 22:25:08.459346 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.459309    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5884778dd9-rdgf9_210f2bf8-3895-4462-9f7b-8811a3785887/telemeter-client/0.log"
Sep 17 22:25:08.497783 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.497763    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5884778dd9-rdgf9_210f2bf8-3895-4462-9f7b-8811a3785887/reload/0.log"
Sep 17 22:25:08.533960 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.533936    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5884778dd9-rdgf9_210f2bf8-3895-4462-9f7b-8811a3785887/kube-rbac-proxy/0.log"
Sep 17 22:25:08.602376 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.602352    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4cbdc4cd-jmdbm_15c4cfd7-f383-46e9-95df-4e1345a67125/thanos-query/0.log"
Sep 17 22:25:08.646164 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.646142    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4cbdc4cd-jmdbm_15c4cfd7-f383-46e9-95df-4e1345a67125/kube-rbac-proxy-web/0.log"
Sep 17 22:25:08.689124 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.689082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4cbdc4cd-jmdbm_15c4cfd7-f383-46e9-95df-4e1345a67125/kube-rbac-proxy/0.log"
Sep 17 22:25:08.733875 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.733851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4cbdc4cd-jmdbm_15c4cfd7-f383-46e9-95df-4e1345a67125/prom-label-proxy/0.log"
Sep 17 22:25:08.776060 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.776038    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4cbdc4cd-jmdbm_15c4cfd7-f383-46e9-95df-4e1345a67125/kube-rbac-proxy-rules/0.log"
Sep 17 22:25:08.810025 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:08.810004    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4cbdc4cd-jmdbm_15c4cfd7-f383-46e9-95df-4e1345a67125/kube-rbac-proxy-metrics/0.log"
Sep 17 22:25:10.878125 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:10.878073    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-7bvxc_a90ed78c-1eae-416d-8d2f-f4195c1c6b08/download-server/0.log"
Sep 17 22:25:11.106016 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.105985    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"]
Sep 17 22:25:11.106326 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.106312    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerName="gather"
Sep 17 22:25:11.106376 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.106328    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerName="gather"
Sep 17 22:25:11.106376 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.106340    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerName="copy"
Sep 17 22:25:11.106376 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.106348    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerName="copy"
Sep 17 22:25:11.106466 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.106398    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerName="gather"
Sep 17 22:25:11.106466 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.106407    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b79c7c-d673-45f2-b0a0-432a572090b7" containerName="copy"
Sep 17 22:25:11.108429 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.108410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.112013 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.111995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fcd62\"/\"default-dockercfg-842mj\""
Sep 17 22:25:11.113271 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.113254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcd62\"/\"kube-root-ca.crt\""
Sep 17 22:25:11.113361 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.113335    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcd62\"/\"openshift-service-ca.crt\""
Sep 17 22:25:11.119334 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.119313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"]
Sep 17 22:25:11.185989 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.185912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-sys\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.185989 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.185943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-podres\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.185989 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.185975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-lib-modules\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.186221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.186064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgkl\" (UniqueName: \"kubernetes.io/projected/96a1b620-a102-4f6f-a133-97a9ca9d119d-kube-api-access-fqgkl\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.186221 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.186098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-proc\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.286565 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-proc\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.286711 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-sys\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.286711 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-podres\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.286711 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-proc\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.286711 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-sys\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.286711 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-lib-modules\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.286711 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgkl\" (UniqueName: \"kubernetes.io/projected/96a1b620-a102-4f6f-a133-97a9ca9d119d-kube-api-access-fqgkl\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.287022 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-podres\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.287022 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.286805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96a1b620-a102-4f6f-a133-97a9ca9d119d-lib-modules\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.295151 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.295130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgkl\" (UniqueName: \"kubernetes.io/projected/96a1b620-a102-4f6f-a133-97a9ca9d119d-kube-api-access-fqgkl\") pod \"perf-node-gather-daemonset-q8n9k\" (UID: \"96a1b620-a102-4f6f-a133-97a9ca9d119d\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.417712 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.417679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:11.686415 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:11.686385    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"]
Sep 17 22:25:11.689572 ip-10-0-139-88 kubenswrapper[2563]: W0917 22:25:11.689545    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod96a1b620_a102_4f6f_a133_97a9ca9d119d.slice/crio-f0188c5aa503bbedb501041c098d031fd8d774466d6648b8b657a0fe4a7abb12 WatchSource:0}: Error finding container f0188c5aa503bbedb501041c098d031fd8d774466d6648b8b657a0fe4a7abb12: Status 404 returned error can't find the container with id f0188c5aa503bbedb501041c098d031fd8d774466d6648b8b657a0fe4a7abb12
Sep 17 22:25:12.178390 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:12.178289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k" event={"ID":"96a1b620-a102-4f6f-a133-97a9ca9d119d","Type":"ContainerStarted","Data":"c91882bf0923b02c2d64871898b2649c6758d147f40b9dcd9522283157ef26f7"}
Sep 17 22:25:12.178390 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:12.178351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k" event={"ID":"96a1b620-a102-4f6f-a133-97a9ca9d119d","Type":"ContainerStarted","Data":"f0188c5aa503bbedb501041c098d031fd8d774466d6648b8b657a0fe4a7abb12"}
Sep 17 22:25:12.178829 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:12.178419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:12.196594 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:12.196541    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k" podStartSLOduration=1.196525751 podStartE2EDuration="1.196525751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:12.194308363 +0000 UTC m=+1618.677794098" watchObservedRunningTime="2026-09-17 22:25:12.196525751 +0000 UTC m=+1618.680011521"
Sep 17 22:25:12.413629 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:12.413599    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x8hln_a214a81a-0083-4cf0-8226-294fcc318e37/dns/0.log"
Sep 17 22:25:12.438352 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:12.438280    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x8hln_a214a81a-0083-4cf0-8226-294fcc318e37/kube-rbac-proxy/0.log"
Sep 17 22:25:12.515091 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:12.515062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-284mc_7ef78f52-2dc7-4839-ae3d-5e8519fde738/dns-node-resolver/0.log"
Sep 17 22:25:13.035024 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:13.034992    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gwsgx_11e30628-ef09-472c-b72e-f068975056ea/node-ca/0.log"
Sep 17 22:25:14.167201 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:14.167163    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-slrhp_34304a54-4fe0-4cbd-888a-d65272625bbf/serve-healthcheck-canary/0.log"
Sep 17 22:25:14.559289 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:14.559266    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24tcl_971a845b-3d43-4ca0-8ad5-c6863b6b497a/kube-rbac-proxy/0.log"
Sep 17 22:25:14.579579 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:14.579546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24tcl_971a845b-3d43-4ca0-8ad5-c6863b6b497a/exporter/0.log"
Sep 17 22:25:14.601513 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:14.601493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24tcl_971a845b-3d43-4ca0-8ad5-c6863b6b497a/extractor/0.log"
Sep 17 22:25:17.022051 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:17.022015    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-7wwqw_dcafe906-ee50-4ca4-b4de-df43d3138008/manager/0.log"
Sep 17 22:25:17.243411 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:17.243379    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xhmk4_a9def44e-a618-4cfe-be84-6cb09323f157/s3-init/0.log"
Sep 17 22:25:17.260925 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:17.260903    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xhmk4_a9def44e-a618-4cfe-be84-6cb09323f157/download-hf-model/0.log"
Sep 17 22:25:18.192460 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:18.192433    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-q8n9k"
Sep 17 22:25:21.521847 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:21.521822    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-k4lqk_341e77d8-ba73-4550-898e-db1c0b2e6faf/migrator/0.log"
Sep 17 22:25:21.540951 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:21.540925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-k4lqk_341e77d8-ba73-4550-898e-db1c0b2e6faf/graceful-termination/0.log"
Sep 17 22:25:22.921096 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:22.921027    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frql5_35891657-8ef8-494c-af51-2d122db69e59/kube-multus-additional-cni-plugins/0.log"
Sep 17 22:25:22.941464 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:22.941436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frql5_35891657-8ef8-494c-af51-2d122db69e59/egress-router-binary-copy/0.log"
Sep 17 22:25:22.962265 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:22.962249    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frql5_35891657-8ef8-494c-af51-2d122db69e59/cni-plugins/0.log"
Sep 17 22:25:22.985626 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:22.985607    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frql5_35891657-8ef8-494c-af51-2d122db69e59/bond-cni-plugin/0.log"
Sep 17 22:25:23.007066 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:23.007046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frql5_35891657-8ef8-494c-af51-2d122db69e59/routeoverride-cni/0.log"
Sep 17 22:25:23.033094 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:23.033073    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frql5_35891657-8ef8-494c-af51-2d122db69e59/whereabouts-cni-bincopy/0.log"
Sep 17 22:25:23.057317 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:23.057302    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frql5_35891657-8ef8-494c-af51-2d122db69e59/whereabouts-cni/0.log"
Sep 17 22:25:23.398035 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:23.398010    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t72bk_7c2c9e9f-9b3a-41c5-9f64-d39e3426a6f6/kube-multus/0.log"
Sep 17 22:25:23.513441 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:23.513402    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r22wg_92cceb95-3224-4dd3-a4e2-d50bf1c58950/network-metrics-daemon/0.log"
Sep 17 22:25:23.533616 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:23.533590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r22wg_92cceb95-3224-4dd3-a4e2-d50bf1c58950/kube-rbac-proxy/0.log"
Sep 17 22:25:24.299368 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.299336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-controller/0.log"
Sep 17 22:25:24.316282 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.316245    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/1.log"
Sep 17 22:25:24.316282 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.316270    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovn-acl-logging/0.log"
Sep 17 22:25:24.338036 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.338014    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/kube-rbac-proxy-node/0.log"
Sep 17 22:25:24.361066 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.361044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 22:25:24.381838 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.381816    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/northd/0.log"
Sep 17 22:25:24.408178 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.408159    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/nbdb/0.log"
Sep 17 22:25:24.435920 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.435899    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/sbdb/0.log"
Sep 17 22:25:24.493841 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:24.493807    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-776jn_16445f01-9a93-4ea6-885e-89db7cfdec04/ovnkube-controller/0.log"
Sep 17 22:25:25.963682 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:25.963654    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ssgcr_a1797ce8-ce06-4cb4-acbc-f253f4d85ef7/network-check-target-container/0.log"
Sep 17 22:25:26.881742 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:26.881714    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-nb27h_f09458f3-a0b3-4662-bf01-84e80a8ebcd6/iptables-alerter/0.log"
Sep 17 22:25:27.614477 ip-10-0-139-88 kubenswrapper[2563]: I0917 22:25:27.614453    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-pkrfb_cacb05e2-766c-4f2d-815d-204ba7bf320e/tuned/0.log"