Sep 17 20:47:07.765656 ip-10-0-139-55 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 20:47:07.765670 ip-10-0-139-55 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 20:47:07.765678 ip-10-0-139-55 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 20:47:07.766105 ip-10-0-139-55 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 20:47:17.857451 ip-10-0-139-55 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 20:47:17.857467 ip-10-0-139-55 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 56c7958bc6b54be8a6bb6ee476855232 --
Sep 17 20:49:18.942619 ip-10-0-139-55 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 20:49:22.095391 ip-10-0-139-55 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:22.095391 ip-10-0-139-55 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 20:49:22.095391 ip-10-0-139-55 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:22.095391 ip-10-0-139-55 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 20:49:22.095391 ip-10-0-139-55 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:22.096047 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.095462    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 20:49:22.098310 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098294    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:22.098310 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098309    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:22.098310 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098313    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098328    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098332    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098335    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098337    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098340    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098343    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098345    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098348    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098351    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098353    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098356    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098358    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098361    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098364    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098366    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098369    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098372    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098376    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:22.098416 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098380    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098386    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098389    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098391    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098394    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098397    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098399    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098402    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098405    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098407    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098410    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098412    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098415    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098418    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098420    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098423    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098426    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098429    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098432    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098434    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:22.098863 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098437    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098439    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098442    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098444    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098447    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098451    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098455    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098459    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098462    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098464    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098467    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098469    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098472    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098475    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098478    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098481    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098484    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098487    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098489    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:22.099356 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098493    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098497    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098500    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098502    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098505    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098508    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098510    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098513    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098515    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098518    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098520    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098523    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098525    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098528    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098530    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098532    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098535    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098537    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098540    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:22.099819 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098542    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098545    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098548    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098550    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098554    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098556    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098559    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098562    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098565    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098568    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098571    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098574    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098576    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098579    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098581    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098584    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098586    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098589    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098592    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.098594    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:22.100271 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099269    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099274    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099277    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099280    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099285    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099289    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099292    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099295    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099298    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099301    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099304    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099307    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099310    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099313    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099334    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099338    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099341    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099343    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099346    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:22.100763 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099349    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099352    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099355    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099358    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099361    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099364    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099366    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099369    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099372    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099374    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099377    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099379    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099382    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099384    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099387    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099389    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099392    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099394    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099397    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099399    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:22.101233 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099402    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099404    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099407    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099410    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099412    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099414    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099417    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099419    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099422    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099426    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099428    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099431    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099433    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099436    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099439    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099442    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099444    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099447    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099449    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099452    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:22.101790 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099454    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099458    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099461    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099464    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099467    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099469    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099472    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099474    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099477    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099479    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099482    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099484    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099486    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099489    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099492    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099494    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099497    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099499    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099502    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099505    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:22.102269 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099508    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099511    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099514    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099516    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099519    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099522    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099525    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099528    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099530    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099532    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099536    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099540    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099543    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099546    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099548    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099551    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099553    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099556    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099558    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:22.102770 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099561    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099924    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099929    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099932    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099935    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099938    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099941    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099944    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099947    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099950    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099952    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099955    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099958    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099962    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099966    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099968    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099971    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099974    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099977    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099979    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:22.103251 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099982    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099984    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099987    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099991    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099995    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.099998    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100001    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100004    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100007    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100009    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100012    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100014    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100017    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100021    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100025    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100028    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100031    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100033    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:22.103805 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100036    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100039    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100041    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100044    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100047    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100049    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100052    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100056    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100059    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100062    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100065    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100068    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100070    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100073    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100075    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100078    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100080    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100083    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100085    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100088    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:22.104367 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100090    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100093    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100096    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100100    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100103    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100106    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100109    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100111    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100114    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100116    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100119    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100121    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100123    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100126    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100128    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100131    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100133    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100136    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100138    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:22.104980 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100142    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100145    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100147    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100150    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100153    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100155    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100158    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100160    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100163    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100165    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100168    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100171    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100173    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100175    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100178    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100181    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100183    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100186    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100188    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100191    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:22.105532 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100193    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100196    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100200    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100270    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100312    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100339    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100343    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100348    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100351    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100355    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100359    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100363    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100366    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100370    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100373    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100376    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100380    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100382    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100385    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100388    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100391    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100395    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100398    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100401    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 20:49:22.106095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100404    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100407    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100411    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100414    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100417    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100420    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100422    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100425    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100428    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100431    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100436    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100440    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100443    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100446    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100449    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100452    2579 flags.go:64] FLAG: --enable-server="true"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100454    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100459    2579 flags.go:64] FLAG: --event-burst="100"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100462    2579 flags.go:64] FLAG: --event-qps="50"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100464    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100468    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100471    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100474    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100477    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100480    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 20:49:22.106800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100488    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100492    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100494    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100497    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100500    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100503    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100505    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100508    2579 flags.go:64] FLAG: --feature-gates=""
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100512    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100515    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100518    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100521    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100524    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100527    2579 flags.go:64] FLAG: --help="false"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100530    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100533    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100535    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100540    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100543    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100546    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100549    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100551    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100554    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 20:49:22.107481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100557    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100560    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100563    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100566    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100569    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100571    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100574    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100577    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100580    2579 flags.go:64] FLAG: --lock-file=""
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100583    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100587    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100590    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100595    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100598    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100601    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100603    2579 flags.go:64] FLAG: --logging-format="text"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100606    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100609    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100612    2579 flags.go:64] FLAG: --manifest-url=""
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100615    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100619    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100622    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100626    2579 flags.go:64] FLAG: --max-pods="110"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100649    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100653    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 20:49:22.108117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100656    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100660    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100663    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100666    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100669    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100676    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100679    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100682    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100685    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100688    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100694    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100696    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100700    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100702    2579 flags.go:64] FLAG: --port="10250"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100705    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100708    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f830bcaaa83e7db7"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100712    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100714    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100718    2579 flags.go:64] FLAG: --register-node="true"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100721    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100724    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100727    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100729    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100732    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100736    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 20:49:22.108766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100738    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100741    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100744    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100746    2579 flags.go:64] FLAG: --runonce="false"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100749    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100752    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100755    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100758    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100762    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100765    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100769    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100772    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100775    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100778    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100780    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100783    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100786    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100789    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100792    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100798    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100801    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100803    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100808    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100810    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100813    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 20:49:22.109455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100816    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100819    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100823    2579 flags.go:64] FLAG: --v="2"
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100826    2579 flags.go:64] FLAG: --version="false"
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100830    2579 flags.go:64] FLAG: --vmodule=""
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100834    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.100837    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100960    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100964    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100967    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100970    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100974    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100977    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100980    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100983    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100987    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100990    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100993    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100996    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.100999    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101002    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:22.110082 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101004    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101008    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101012    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101015    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101017    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101021    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101024    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101026    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101029    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101032    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101034    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101037    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101039    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101042    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101045    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101047    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101050    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101053    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101055    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101058    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:22.110586 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101061    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101063    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101066    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101068    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101071    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101073    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101076    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101079    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101082    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101084    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101087    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101089    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101092    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101094    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101097    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101099    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101102    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101106    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101109    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101111    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:22.111075 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101114    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101116    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101119    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101121    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101124    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101126    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101129    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101132    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101135    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101138    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101140    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101143    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101145    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101148    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101150    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101152    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101155    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101157    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101160    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101163    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:22.111608 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101165    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101168    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101170    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101173    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101175    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101179    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101183    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101186    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101188    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101193    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101196    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101199    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101201    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101204    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101206    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101209    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101212    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101216    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:22.112105 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101219    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101226    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101229    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101231    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101234    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101237    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.101239    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.101245    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.108485    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.108499    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.108519    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.108528    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108580    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108585    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:22.112553 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108589    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108593    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108596    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108599    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108602    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108604    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108607    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108610    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108612    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108615    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108618    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108621    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108623    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108626    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108628    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108631    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108634    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108636    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108639    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108641    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108644    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:22.112905 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108646    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108650    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108653    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108656    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108659    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108663    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108666    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108669    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108672    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108676    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108679    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108682    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108684    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108687    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108689    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108692    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108695    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108697    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108700    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:22.113499 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108702    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108705    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108707    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108710    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108712    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108715    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108717    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108720    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108722    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108725    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108727    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108730    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108732    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108735    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108737    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108740    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108742    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108745    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108747    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108751    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:22.114000 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108754    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108757    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108760    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108762    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108765    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108768    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108771    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108774    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108777    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108780    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108782    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108785    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108788    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108790    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108793    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108795    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108797    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108800    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108803    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:22.114489 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108805    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108808    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108810    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108813    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108815    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108818    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108820    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108823    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108825    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108828    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108831    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108833    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108837    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108840    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108843    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108846    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108848    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:22.114947 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108851    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.108856    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108966    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108971    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108974    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108976    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108979    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108981    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108984    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108986    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108989    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108991    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108994    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108996    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.108999    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:22.115525 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109002    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109004    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109007    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109009    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109012    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109014    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109017    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109019    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109022    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109026    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109030    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109033    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109036    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109039    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109042    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109044    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109047    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109050    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109052    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:22.116273 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109054    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109057    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109059    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109062    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109065    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109067    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109070    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109072    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109074    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109077    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109080    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109082    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109085    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109087    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109089    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109092    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109094    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109097    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109099    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109102    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:22.116752 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109104    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109107    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109109    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109112    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109114    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109117    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109120    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109122    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109125    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109128    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109131    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109133    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109136    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109138    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109141    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109143    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109145    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109148    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109150    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109153    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:22.117266 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109155    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109158    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109160    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109163    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109165    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109168    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109170    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109173    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109175    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109178    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109180    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109182    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109185    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109187    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109190    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109192    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109194    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109197    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109200    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109202    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:22.117881 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109205    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109207    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109211    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109215    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109217    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109219    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:22.109223    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.109228    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.109335    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.111190    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.112021    2579 server.go:1019] "Starting client certificate rotation"
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.112112    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.112147    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:22.118375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.114006    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:22.118699 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.115052    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:22.127676 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.127659    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 17 20:49:22.129543 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.129532    2579 log.go:25] "Validated CRI v1 image API"
Sep 17 20:49:22.130332 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.130302    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 20:49:22.132245 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.132223    2579 fs.go:135] Filesystem UUIDs: map[4740df33-db6e-4bf7-a127-619d56d638b8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 db8e0e46-6e85-4a1b-b3ff-459a47d0f17a:/dev/nvme0n1p3]
Sep 17 20:49:22.132336 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.132244    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 20:49:22.137063 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.136948    2579 manager.go:216] Machine: {Timestamp:2026-09-17 20:49:22.135841405 +0000 UTC m=+1.829650747 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097039 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec289145dc358a43a83510ccd95aca45 SystemUUID:ec289145-dc35-8a43-a835-10ccd95aca45 BootID:56c7958b-c6b5-4be8-a6bb-6ee476855232 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:2d:c1:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:2d:c1:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:af:52:e3:ad:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 20:49:22.137063 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.137051    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 20:49:22.137222 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.137151    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 20:49:22.138569 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.138542    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 20:49:22.138723 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.138571    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 20:49:22.139623 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.139611    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 20:49:22.139685 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.139630    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 20:49:22.139685 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.139640    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:22.139685 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.139653    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:22.139817 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.139737    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 20:49:22.144084 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144070    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:22.144210 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144199    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 20:49:22.144269 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144240    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 20:49:22.144269 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144256    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 20:49:22.144382 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144276    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 20:49:22.144382 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144292    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 17 20:49:22.144382 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144304    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 20:49:22.144687 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144672    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:22.144752 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.144697    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:22.146546 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.146530    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 20:49:22.148615 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.148602    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 20:49:22.148654 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.148633    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 20:49:22.149681 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149670    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 20:49:22.149714 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149686    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 20:49:22.149714 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149693    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 20:49:22.149714 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149698    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 20:49:22.149714 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149707    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 20:49:22.149714 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149712    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149727    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149732    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149739    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149745    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149753    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149758    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149776    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 20:49:22.149851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.149781    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 20:49:22.153761 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.153743    2579 server.go:1268] "Started kubelet"
Sep 17 20:49:22.153928 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.153900    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 20:49:22.153986 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.153926    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 20:49:22.153986 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.153947    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 20:49:22.154461 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.154441    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:22.154664 ip-10-0-139-55 systemd[1]: Started Kubernetes Kubelet.
Sep 17 20:49:22.184034 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.184007    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 20:49:22.185090 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.185068    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 20:49:22.185299 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.185277    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 20:49:22.185553 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.185439    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 20:49:22.185553 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.185442    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 20:49:22.185713 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.185697    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 20:49:22.186711 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.186683    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6wfb"
Sep 17 20:49:22.190519 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.189760    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-55.ec2.internal.18d636ce6d724522  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-55.ec2.internal,UID:ip-10-0-139-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-55.ec2.internal,},FirstTimestamp:2026-09-17 20:49:22.153719074 +0000 UTC m=+1.847528400,LastTimestamp:2026-09-17 20:49:22.153719074 +0000 UTC m=+1.847528400,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-55.ec2.internal,}"
Sep 17 20:49:22.191776 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.191756    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 20:49:22.191863 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.191776    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:22.191863 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.191854    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.192386 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192363    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 20:49:22.192386 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192386    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 20:49:22.192535 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192496    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 20:49:22.192535 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192505    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 20:49:22.192535 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192521    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 20:49:22.192713 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192699    2579 factory.go:55] Registering systemd factory
Sep 17 20:49:22.192765 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192718    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 17 20:49:22.192916 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192896    2579 factory.go:153] Registering CRI-O factory
Sep 17 20:49:22.192916 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192910    2579 factory.go:223] Registration of the crio container factory successfully
Sep 17 20:49:22.192994 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192958    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 20:49:22.192994 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192973    2579 factory.go:103] Registering Raw factory
Sep 17 20:49:22.192994 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.192987    2579 manager.go:1196] Started watching for new ooms in manager
Sep 17 20:49:22.193338 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.193309    2579 manager.go:349] Starting recovery of all containers
Sep 17 20:49:22.194231 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.194210    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6wfb"
Sep 17 20:49:22.202679 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.202559    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:22.203907 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.203892    2579 manager.go:354] Recovery completed
Sep 17 20:49:22.207014 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.206999    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-55.ec2.internal\" not found" node="ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.208501 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.208490    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:22.210283 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.210268    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:22.210374 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.210298    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:22.210374 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.210308    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:22.210844 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.210829    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 20:49:22.210844 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.210842    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 20:49:22.210950 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.210831    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:22.210950 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.210861    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:22.211556 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.211543    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:22.211607 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.211567    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:22.211607 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.211577    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:22.212848 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.212837    2579 policy_none.go:49] "None policy: Start"
Sep 17 20:49:22.212883 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.212854    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 20:49:22.212883 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.212863    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 20:49:22.214766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.214756    2579 policy_none.go:47] "Start"
Sep 17 20:49:22.269423 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.269410    2579 manager.go:337] "Starting Device Plugin manager"
Sep 17 20:49:22.269523 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.269447    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 20:49:22.269523 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.269462    2579 server.go:87] "Starting device plugin registration server"
Sep 17 20:49:22.269732 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.269720    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 20:49:22.269781 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.269740    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 20:49:22.269849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.269834    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 20:49:22.269934 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.269924    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 20:49:22.269985 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.269936    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 20:49:22.270470 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.270451    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 20:49:22.270539 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.270501    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.291677 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.291652    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 20:49:22.293021 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.293000    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 20:49:22.293097 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.293041    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 20:49:22.293097 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.293059    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 20:49:22.294890 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.293371    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 20:49:22.295858 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.295840    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:22.369935 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.369866    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:22.370797 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.370775    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:22.370883 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.370802    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:22.370883 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.370812    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:22.370883 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.370835    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.379223 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.379205    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.379310 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.379233    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-55.ec2.internal\": node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.394287 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.394248    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"]
Sep 17 20:49:22.394379 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.394336    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:22.395116 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.395100    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:22.395181 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.395126    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:22.395181 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.395138    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:22.397564 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.397552    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:22.397685 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.397673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.397723 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.397701    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:22.399502 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.399480    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:22.399502 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.399488    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:22.399603 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.399510    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:22.399603 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.399511    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:22.399603 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.399537    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:22.399603 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.399525    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:22.401880 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.401864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.401955 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.401890    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:22.402631 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.402613    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:22.402710 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.402648    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:22.402710 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.402662    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:22.433840 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.433820    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-55.ec2.internal\" not found" node="ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.438174 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.438157    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-55.ec2.internal\" not found" node="ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.445434 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.445414    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.546117 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.546096    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.594543 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.594522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e878fac6c20a59b3277411748de820b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-55.ec2.internal\" (UID: \"3e878fac6c20a59b3277411748de820b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.594627 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.594546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.594627 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.594562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.646492 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.646433    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.695680 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.695656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.695759 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.695684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e878fac6c20a59b3277411748de820b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-55.ec2.internal\" (UID: \"3e878fac6c20a59b3277411748de820b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.695759 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.695699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.695759 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.695748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.695852 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.695771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.695852 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.695771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e878fac6c20a59b3277411748de820b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-55.ec2.internal\" (UID: \"3e878fac6c20a59b3277411748de820b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.736871 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.736854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.742160 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:22.742005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:22.747067 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.747050    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.847756 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.847721    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:22.948237 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:22.948173    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:23.048965 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:23.048937    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:23.112476 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.112448    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 20:49:23.113052 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.112615    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:23.113052 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.112626    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:23.150010 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:23.149986    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 17 20:49:23.162655 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.162636    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:23.192444 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.192419    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:23.192444 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.192439    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:23.200303 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.200247    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:44:22 +0000 UTC" deadline="2028-03-28 20:46:37.782801291 +0000 UTC"
Sep 17 20:49:23.200303 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.200272    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13391h57m14.582531781s"
Sep 17 20:49:23.203808 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.203793    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:23.205172 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.205150    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 17 20:49:23.206613 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.206592    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:23.215441 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.215420    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:23.235797 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.235774    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2c9q"
Sep 17 20:49:23.242847 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.242832    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2c9q"
Sep 17 20:49:23.330098 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:23.330071    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e878fac6c20a59b3277411748de820b.slice/crio-31d5daa5c24c1cfd593acedd766924af88856ca947fad9159a1495badfa4b0ab WatchSource:0}: Error finding container 31d5daa5c24c1cfd593acedd766924af88856ca947fad9159a1495badfa4b0ab: Status 404 returned error can't find the container with id 31d5daa5c24c1cfd593acedd766924af88856ca947fad9159a1495badfa4b0ab
Sep 17 20:49:23.330435 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:23.330415    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f37f127c93297148e7dbac392282653.slice/crio-acb77aa8454e48c17ab79ccb6d272ea04bcdd3e55d4f13995131ad139c5faccd WatchSource:0}: Error finding container acb77aa8454e48c17ab79ccb6d272ea04bcdd3e55d4f13995131ad139c5faccd: Status 404 returned error can't find the container with id acb77aa8454e48c17ab79ccb6d272ea04bcdd3e55d4f13995131ad139c5faccd
Sep 17 20:49:23.334838 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.334826    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:49:23.436575 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:23.436548    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:24.113258 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.113076    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:24.145758 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.145730    2579 apiserver.go:52] "Watching apiserver"
Sep 17 20:49:24.151777 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.151754    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 20:49:24.152293 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.152274    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j","openshift-cluster-node-tuning-operator/tuned-snzhr","openshift-dns/node-resolver-xr4c8","openshift-multus/multus-76468","openshift-multus/multus-additional-cni-plugins-4sftt","openshift-multus/network-metrics-daemon-sn8wn","kube-system/global-pull-secret-syncer-x66tf","openshift-image-registry/node-ca-cspdc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal","openshift-network-diagnostics/network-check-target-r5xpp","openshift-network-operator/iptables-alerter-ss6d8","openshift-ovn-kubernetes/ovnkube-node-khg2b","kube-system/konnectivity-agent-t6tpr","kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"]
Sep 17 20:49:24.155044 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.155019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.155134 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.155104    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:24.157030 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.156993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.160682 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.160662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 20:49:24.160789 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.160772    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 20:49:24.160851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.160663    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4l9rs\""
Sep 17 20:49:24.161007 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.160989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.162911 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.162892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76468"
Sep 17 20:49:24.162985 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.162918    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 20:49:24.163571 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.163547    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 20:49:24.164592 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.163689    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wtrqz\""
Sep 17 20:49:24.165155 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.164898    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 20:49:24.166073 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.165348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 20:49:24.166073 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.165378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8pwr6\""
Sep 17 20:49:24.166073 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.165439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.166073 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.165574    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 20:49:24.166073 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.165765    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 20:49:24.167563 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.167539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:24.167721 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.167676    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:24.167797 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.167724    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 20:49:24.167797 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.167742    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 20:49:24.168029 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.168012    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wrn9l\""
Sep 17 20:49:24.170478 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.170463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.172478 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.172459    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 20:49:24.172669 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.172647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 20:49:24.172791 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.172774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7ck58\""
Sep 17 20:49:24.173025 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.173007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.173147 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.172466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 20:49:24.175342 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.175303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.175706 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.175692    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 20:49:24.175921 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.175900    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gpxh2\""
Sep 17 20:49:24.175991 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.175800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 20:49:24.176135 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.175846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 20:49:24.177493 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.177165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:24.177493 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.177225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:24.177493 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.177395    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:24.177678 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.177506    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:24.177678 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.177585    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjlxj\""
Sep 17 20:49:24.180462 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.179918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.182093 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.182063    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:24.182501 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.182434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.184100 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.184081    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:24.184185 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.184115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ndhk6\""
Sep 17 20:49:24.184383 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.184367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 20:49:24.185187 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.185168    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 20:49:24.185462 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.185441    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 20:49:24.185552 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.185492    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 20:49:24.185650 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.185634    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 20:49:24.185721 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.185706    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 20:49:24.185890 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.185875    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m5c6c\""
Sep 17 20:49:24.186184 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.186100    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 20:49:24.194891 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.194873    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 20:49:24.204718 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-kubelet\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.204784 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysctl-conf\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.204839 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-systemd-units\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.204839 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-ovn\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.204941 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-log-socket\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.204941 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/336e56a1-cbd3-497b-b784-4890ae07845a-ovn-node-metrics-cert\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.204941 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-cni-bin\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.205081 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-conf-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.205081 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-system-cni-dir\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.205081 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.204998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-registration-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.205081 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-kubelet\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.205081 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-run-netns\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.205081 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.205388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8dd712c9-6a04-4eeb-abe9-bc6f45dece63-agent-certs\") pod \"konnectivity-agent-t6tpr\" (UID: \"8dd712c9-6a04-4eeb-abe9-bc6f45dece63\") " pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.205388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-systemd\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.205388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-tmp-dir\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.205388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.205388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.205388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.205388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-cni-netd\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8dd712c9-6a04-4eeb-abe9-bc6f45dece63-konnectivity-ca\") pod \"konnectivity-agent-t6tpr\" (UID: \"8dd712c9-6a04-4eeb-abe9-bc6f45dece63\") " pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-var-lib-kubelet\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwnw\" (UniqueName: \"kubernetes.io/projected/bf9e641a-5778-4acd-aea2-d08a1b8da14a-kube-api-access-cqwnw\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-host-slash\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcrhc\" (UniqueName: \"kubernetes.io/projected/336e56a1-cbd3-497b-b784-4890ae07845a-kube-api-access-zcrhc\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-system-cni-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-etc-kubernetes\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-kubernetes\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-sys-fs\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.205720 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvskw\" (UniqueName: \"kubernetes.io/projected/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-kube-api-access-fvskw\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf9e641a-5778-4acd-aea2-d08a1b8da14a-tmp\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-iptables-alerter-script\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cnibin\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkmfv\" (UniqueName: \"kubernetes.io/projected/410d7742-1dd0-4ca9-abcf-3b999c572cbf-kube-api-access-nkmfv\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.205968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-systemd\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-sys\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.206132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2p2\" (UniqueName: \"kubernetes.io/projected/720792d8-c7ab-4c51-8ce1-7a79a051e9af-kube-api-access-5r2p2\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-cni-bin\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-cni-multus\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysconfig\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdgd\" (UniqueName: \"kubernetes.io/projected/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-kube-api-access-krdgd\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-socket-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-socket-dir-parent\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-tuned\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d99s\" (UniqueName: \"kubernetes.io/projected/2193e970-4de6-43f0-ab19-e94574c01546-kube-api-access-5d99s\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-run-ovn-kubernetes\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-netns\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.206531 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-host\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-ovnkube-config\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2193e970-4de6-43f0-ab19-e94574c01546-host\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-daemon-config\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysctl-d\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-hosts-file\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2193e970-4de6-43f0-ab19-e94574c01546-serviceca\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-etc-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-cnibin\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-hostroot\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-modprobe-d\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.206969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-env-overrides\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-k8s-cni-cncf-io\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-lib-modules\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-run\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.207128 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2bm\" (UniqueName: \"kubernetes.io/projected/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-kube-api-access-xc2bm\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-device-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-etc-selinux\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-slash\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-var-lib-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-multus-certs\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-kubelet-config\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-dbus\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpc2\" (UniqueName: \"kubernetes.io/projected/03f9073e-d15c-4ce7-8a29-149b0fde2791-kube-api-access-dhpc2\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-node-log\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-ovnkube-script-lib\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-cni-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.207972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.207463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-cni-binary-copy\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.245616 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.245555    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:23 +0000 UTC" deadline="2028-04-30 17:25:27.002492312 +0000 UTC"
Sep 17 20:49:24.245616 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.245619    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14180h36m2.756880079s"
Sep 17 20:49:24.298429 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.298372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal" event={"ID":"3e878fac6c20a59b3277411748de820b","Type":"ContainerStarted","Data":"31d5daa5c24c1cfd593acedd766924af88856ca947fad9159a1495badfa4b0ab"}
Sep 17 20:49:24.299604 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.299571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" event={"ID":"7f37f127c93297148e7dbac392282653","Type":"ContainerStarted","Data":"acb77aa8454e48c17ab79ccb6d272ea04bcdd3e55d4f13995131ad139c5faccd"}
Sep 17 20:49:24.307761 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysctl-d\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-hosts-file\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2193e970-4de6-43f0-ab19-e94574c01546-serviceca\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-etc-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-cnibin\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-hostroot\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-modprobe-d\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-hosts-file\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.307901 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-env-overrides\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-k8s-cni-cncf-io\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-cnibin\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-lib-modules\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysctl-d\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-run\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.307979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2bm\" (UniqueName: \"kubernetes.io/projected/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-kube-api-access-xc2bm\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-device-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-etc-selinux\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-hostroot\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-slash\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-lib-modules\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-var-lib-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-var-lib-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-multus-certs\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-modprobe-d\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-kubelet-config\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-k8s-cni-cncf-io\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.308387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-dbus\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpc2\" (UniqueName: \"kubernetes.io/projected/03f9073e-d15c-4ce7-8a29-149b0fde2791-kube-api-access-dhpc2\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-node-log\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-ovnkube-script-lib\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-multus-certs\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-kubelet-config\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-etc-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-cni-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-dbus\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-cni-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-run\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-cni-binary-copy\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-kubelet\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-node-log\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysctl-conf\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-systemd-units\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-ovn\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-log-socket\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.309200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-env-overrides\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/336e56a1-cbd3-497b-b784-4890ae07845a-ovn-node-metrics-cert\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-cni-bin\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-conf-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-system-cni-dir\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-registration-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2193e970-4de6-43f0-ab19-e94574c01546-serviceca\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-device-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-cni-bin\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-conf-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-system-cni-dir\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysctl-conf\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-registration-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-slash\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-kubelet\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-etc-selinux\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.310098 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-run-netns\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-kubelet\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-kubelet\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-systemd-units\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8dd712c9-6a04-4eeb-abe9-bc6f45dece63-agent-certs\") pod \"konnectivity-agent-t6tpr\" (UID: \"8dd712c9-6a04-4eeb-abe9-bc6f45dece63\") " pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-run-netns\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-systemd\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-ovn\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.308993    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-tmp-dir\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-log-socket\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-cni-netd\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8dd712c9-6a04-4eeb-abe9-bc6f45dece63-konnectivity-ca\") pod \"konnectivity-agent-t6tpr\" (UID: \"8dd712c9-6a04-4eeb-abe9-bc6f45dece63\") " pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-ovnkube-script-lib\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.310885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-var-lib-kubelet\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-openvswitch\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwnw\" (UniqueName: \"kubernetes.io/projected/bf9e641a-5778-4acd-aea2-d08a1b8da14a-kube-api-access-cqwnw\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-host-slash\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcrhc\" (UniqueName: \"kubernetes.io/projected/336e56a1-cbd3-497b-b784-4890ae07845a-kube-api-access-zcrhc\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-system-cni-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-etc-kubernetes\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-systemd\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-kubernetes\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-sys-fs\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-kubernetes\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-cni-netd\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvskw\" (UniqueName: \"kubernetes.io/projected/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-kube-api-access-fvskw\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf9e641a-5778-4acd-aea2-d08a1b8da14a-tmp\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-var-lib-kubelet\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-iptables-alerter-script\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.311762 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-cni-binary-copy\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cnibin\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkmfv\" (UniqueName: \"kubernetes.io/projected/410d7742-1dd0-4ca9-abcf-3b999c572cbf-kube-api-access-nkmfv\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-systemd\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-tmp-dir\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-system-cni-dir\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-host-slash\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.309797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cnibin\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.309813    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.309924    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret podName:1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.80990223 +0000 UTC m=+4.503711562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret") pod "global-pull-secret-syncer-x66tf" (UID: "1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-etc-kubernetes\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-sys\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.312418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-run-systemd\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2p2\" (UniqueName: \"kubernetes.io/projected/720792d8-c7ab-4c51-8ce1-7a79a051e9af-kube-api-access-5r2p2\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-cni-bin\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-sys\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8dd712c9-6a04-4eeb-abe9-bc6f45dece63-konnectivity-ca\") pod \"konnectivity-agent-t6tpr\" (UID: \"8dd712c9-6a04-4eeb-abe9-bc6f45dece63\") " pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-cni-multus\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysconfig\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-cni-bin\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdgd\" (UniqueName: \"kubernetes.io/projected/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-kube-api-access-krdgd\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-iptables-alerter-script\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-socket-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-var-lib-cni-multus\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-sysconfig\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-socket-dir-parent\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-tuned\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-sys-fs\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.313078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d99s\" (UniqueName: \"kubernetes.io/projected/2193e970-4de6-43f0-ab19-e94574c01546-kube-api-access-5d99s\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-socket-dir-parent\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-run-ovn-kubernetes\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-netns\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-host\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.310866    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.310920    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.810905456 +0000 UTC m=+4.504714779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-host-run-netns\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336e56a1-cbd3-497b-b784-4890ae07845a-host-run-ovn-kubernetes\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.310974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf9e641a-5778-4acd-aea2-d08a1b8da14a-host\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-ovnkube-config\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2193e970-4de6-43f0-ab19-e94574c01546-host\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-daemon-config\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/720792d8-c7ab-4c51-8ce1-7a79a051e9af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2193e970-4de6-43f0-ab19-e94574c01546-host\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.313666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/336e56a1-cbd3-497b-b784-4890ae07845a-ovnkube-config\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.314225 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-multus-daemon-config\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.314225 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.311784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03f9073e-d15c-4ce7-8a29-149b0fde2791-socket-dir\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.314225 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.312853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/336e56a1-cbd3-497b-b784-4890ae07845a-ovn-node-metrics-cert\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.314225 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.312991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8dd712c9-6a04-4eeb-abe9-bc6f45dece63-agent-certs\") pod \"konnectivity-agent-t6tpr\" (UID: \"8dd712c9-6a04-4eeb-abe9-bc6f45dece63\") " pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.314225 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.313792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf9e641a-5778-4acd-aea2-d08a1b8da14a-tmp\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.316173 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.316147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf9e641a-5778-4acd-aea2-d08a1b8da14a-etc-tuned\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.316375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.316356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpc2\" (UniqueName: \"kubernetes.io/projected/03f9073e-d15c-4ce7-8a29-149b0fde2791-kube-api-access-dhpc2\") pod \"aws-ebs-csi-driver-node-c969j\" (UID: \"03f9073e-d15c-4ce7-8a29-149b0fde2791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.316913 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.316892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2bm\" (UniqueName: \"kubernetes.io/projected/b0f2f00d-88fd-4ecb-adb7-a1cc9836d870-kube-api-access-xc2bm\") pod \"node-resolver-xr4c8\" (UID: \"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870\") " pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.319548 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.319527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwnw\" (UniqueName: \"kubernetes.io/projected/bf9e641a-5778-4acd-aea2-d08a1b8da14a-kube-api-access-cqwnw\") pod \"tuned-snzhr\" (UID: \"bf9e641a-5778-4acd-aea2-d08a1b8da14a\") " pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.323458 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.323426    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:24.323458 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.323448    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:24.323458 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.323461    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt68w for pod openshift-network-diagnostics/network-check-target-r5xpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:24.323646 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.323510    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w podName:3df1144b-3aa8-4688-aa04-2e16ff789302 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.823495903 +0000 UTC m=+4.517305231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dt68w" (UniqueName: "kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w") pod "network-check-target-r5xpp" (UID: "3df1144b-3aa8-4688-aa04-2e16ff789302") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:24.323646 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.323565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcrhc\" (UniqueName: \"kubernetes.io/projected/336e56a1-cbd3-497b-b784-4890ae07845a-kube-api-access-zcrhc\") pod \"ovnkube-node-khg2b\" (UID: \"336e56a1-cbd3-497b-b784-4890ae07845a\") " pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.325155 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.325128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d99s\" (UniqueName: \"kubernetes.io/projected/2193e970-4de6-43f0-ab19-e94574c01546-kube-api-access-5d99s\") pod \"node-ca-cspdc\" (UID: \"2193e970-4de6-43f0-ab19-e94574c01546\") " pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.325769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.325730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvskw\" (UniqueName: \"kubernetes.io/projected/3d96d7a2-ce04-4878-9f9f-e8f3ec947d56-kube-api-access-fvskw\") pod \"multus-76468\" (UID: \"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56\") " pod="openshift-multus/multus-76468"
Sep 17 20:49:24.326624 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.326595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdgd\" (UniqueName: \"kubernetes.io/projected/0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85-kube-api-access-krdgd\") pod \"iptables-alerter-ss6d8\" (UID: \"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85\") " pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.327244 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.327184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkmfv\" (UniqueName: \"kubernetes.io/projected/410d7742-1dd0-4ca9-abcf-3b999c572cbf-kube-api-access-nkmfv\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:24.327444 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.327421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2p2\" (UniqueName: \"kubernetes.io/projected/720792d8-c7ab-4c51-8ce1-7a79a051e9af-kube-api-access-5r2p2\") pod \"multus-additional-cni-plugins-4sftt\" (UID: \"720792d8-c7ab-4c51-8ce1-7a79a051e9af\") " pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.470289 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.470187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:24.479190 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.479161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76468"
Sep 17 20:49:24.485798 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.485770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xr4c8"
Sep 17 20:49:24.494301 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.494279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4sftt"
Sep 17 20:49:24.501070 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.501055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cspdc"
Sep 17 20:49:24.509483 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.509469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j"
Sep 17 20:49:24.515179 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.515159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-snzhr"
Sep 17 20:49:24.522168 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.522151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ss6d8"
Sep 17 20:49:24.527824 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.527807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:24.669152 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.669117    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:24.815410 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.815333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:24.815410 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.815390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:24.815666 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.815438    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:24.815666 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.815478    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.815666 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.815518    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:25.815495946 +0000 UTC m=+5.509305269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:24.815666 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.815539    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret podName:1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:25.815529148 +0000 UTC m=+5.509338456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret") pod "global-pull-secret-syncer-x66tf" (UID: "1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.915970 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:24.915931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:24.916133 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.916110    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:24.916216 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.916138    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:24.916216 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.916149    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt68w for pod openshift-network-diagnostics/network-check-target-r5xpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:24.916216 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:24.916205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w podName:3df1144b-3aa8-4688-aa04-2e16ff789302 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:25.916188434 +0000 UTC m=+5.609997746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt68w" (UniqueName: "kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w") pod "network-check-target-r5xpp" (UID: "3df1144b-3aa8-4688-aa04-2e16ff789302") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:25.169303 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.169279    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd712c9_6a04_4eeb_abe9_bc6f45dece63.slice/crio-bfbfb975698b61338c1147253e31c53a65d1b1cb651b84abe755655ff9f89254 WatchSource:0}: Error finding container bfbfb975698b61338c1147253e31c53a65d1b1cb651b84abe755655ff9f89254: Status 404 returned error can't find the container with id bfbfb975698b61338c1147253e31c53a65d1b1cb651b84abe755655ff9f89254
Sep 17 20:49:25.172584 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.172557    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720792d8_c7ab_4c51_8ce1_7a79a051e9af.slice/crio-02ee7f8bf651b457f124f5454df166d078c25725313eeac1b3c677461b085d8f WatchSource:0}: Error finding container 02ee7f8bf651b457f124f5454df166d078c25725313eeac1b3c677461b085d8f: Status 404 returned error can't find the container with id 02ee7f8bf651b457f124f5454df166d078c25725313eeac1b3c677461b085d8f
Sep 17 20:49:25.174567 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.174542    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f2f00d_88fd_4ecb_adb7_a1cc9836d870.slice/crio-bc979ef22739dc726132d292e739941a1be4924772f7f500b34d61cfcf548f10 WatchSource:0}: Error finding container bc979ef22739dc726132d292e739941a1be4924772f7f500b34d61cfcf548f10: Status 404 returned error can't find the container with id bc979ef22739dc726132d292e739941a1be4924772f7f500b34d61cfcf548f10
Sep 17 20:49:25.175637 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.175616    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336e56a1_cbd3_497b_b784_4890ae07845a.slice/crio-282c084efe02f020873b6f287f162e48d6bb0d496b1454c443e28ec1e72d2342 WatchSource:0}: Error finding container 282c084efe02f020873b6f287f162e48d6bb0d496b1454c443e28ec1e72d2342: Status 404 returned error can't find the container with id 282c084efe02f020873b6f287f162e48d6bb0d496b1454c443e28ec1e72d2342
Sep 17 20:49:25.177243 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.177219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9e641a_5778_4acd_aea2_d08a1b8da14a.slice/crio-32c762d8ebcd1499c0480e6c34ce323b77a56c0920dd0a61c4d62e14e366185d WatchSource:0}: Error finding container 32c762d8ebcd1499c0480e6c34ce323b77a56c0920dd0a61c4d62e14e366185d: Status 404 returned error can't find the container with id 32c762d8ebcd1499c0480e6c34ce323b77a56c0920dd0a61c4d62e14e366185d
Sep 17 20:49:25.178296 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.178273    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0815ce6c_2ceb_4a9a_9c15_abd5cdb36c85.slice/crio-8b44ccf03f8dda693cf8bdacd3951e8ced9c35c57de5c48b0b558aef110f942a WatchSource:0}: Error finding container 8b44ccf03f8dda693cf8bdacd3951e8ced9c35c57de5c48b0b558aef110f942a: Status 404 returned error can't find the container with id 8b44ccf03f8dda693cf8bdacd3951e8ced9c35c57de5c48b0b558aef110f942a
Sep 17 20:49:25.178850 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.178828    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f9073e_d15c_4ce7_8a29_149b0fde2791.slice/crio-eff1e3a44a0a3e07aaa318a0ed7a42f382a21cdb1fa741b0ebbd384d53042a0f WatchSource:0}: Error finding container eff1e3a44a0a3e07aaa318a0ed7a42f382a21cdb1fa741b0ebbd384d53042a0f: Status 404 returned error can't find the container with id eff1e3a44a0a3e07aaa318a0ed7a42f382a21cdb1fa741b0ebbd384d53042a0f
Sep 17 20:49:25.180543 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.180498    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2193e970_4de6_43f0_ab19_e94574c01546.slice/crio-fc1b0698e7cdd696e1b5495e90b0f39745214a59f5e43df4e19ae22af8c16f03 WatchSource:0}: Error finding container fc1b0698e7cdd696e1b5495e90b0f39745214a59f5e43df4e19ae22af8c16f03: Status 404 returned error can't find the container with id fc1b0698e7cdd696e1b5495e90b0f39745214a59f5e43df4e19ae22af8c16f03
Sep 17 20:49:25.182006 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:25.181254    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d96d7a2_ce04_4878_9f9f_e8f3ec947d56.slice/crio-7970e47c14953696acb5a380ebf150f02ab620c04dca7277558aa3676665da10 WatchSource:0}: Error finding container 7970e47c14953696acb5a380ebf150f02ab620c04dca7277558aa3676665da10: Status 404 returned error can't find the container with id 7970e47c14953696acb5a380ebf150f02ab620c04dca7277558aa3676665da10
Sep 17 20:49:25.246694 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.246653    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:23 +0000 UTC" deadline="2028-07-03 11:50:26.979335652 +0000 UTC"
Sep 17 20:49:25.246694 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.246692    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15711h1m1.73264565s"
Sep 17 20:49:25.293508 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.293480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:25.293633 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.293480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:25.293633 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.293591    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:25.293745 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.293480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:25.293745 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.293681    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:25.293838 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.293779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:25.302230 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.302208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76468" event={"ID":"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56","Type":"ContainerStarted","Data":"7970e47c14953696acb5a380ebf150f02ab620c04dca7277558aa3676665da10"}
Sep 17 20:49:25.303536 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.303514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ss6d8" event={"ID":"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85","Type":"ContainerStarted","Data":"8b44ccf03f8dda693cf8bdacd3951e8ced9c35c57de5c48b0b558aef110f942a"}
Sep 17 20:49:25.304801 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.304778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerStarted","Data":"02ee7f8bf651b457f124f5454df166d078c25725313eeac1b3c677461b085d8f"}
Sep 17 20:49:25.306721 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.306700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal" event={"ID":"3e878fac6c20a59b3277411748de820b","Type":"ContainerStarted","Data":"04598b0a700f914032977861894edc7072134103ada08d9167475b3b29a2f6b7"}
Sep 17 20:49:25.307925 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.307898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cspdc" event={"ID":"2193e970-4de6-43f0-ab19-e94574c01546","Type":"ContainerStarted","Data":"fc1b0698e7cdd696e1b5495e90b0f39745214a59f5e43df4e19ae22af8c16f03"}
Sep 17 20:49:25.309122 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.309100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j" event={"ID":"03f9073e-d15c-4ce7-8a29-149b0fde2791","Type":"ContainerStarted","Data":"eff1e3a44a0a3e07aaa318a0ed7a42f382a21cdb1fa741b0ebbd384d53042a0f"}
Sep 17 20:49:25.310220 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.310200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-snzhr" event={"ID":"bf9e641a-5778-4acd-aea2-d08a1b8da14a","Type":"ContainerStarted","Data":"32c762d8ebcd1499c0480e6c34ce323b77a56c0920dd0a61c4d62e14e366185d"}
Sep 17 20:49:25.311418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.311389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"282c084efe02f020873b6f287f162e48d6bb0d496b1454c443e28ec1e72d2342"}
Sep 17 20:49:25.313025 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.313004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xr4c8" event={"ID":"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870","Type":"ContainerStarted","Data":"bc979ef22739dc726132d292e739941a1be4924772f7f500b34d61cfcf548f10"}
Sep 17 20:49:25.314030 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.314011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6tpr" event={"ID":"8dd712c9-6a04-4eeb-abe9-bc6f45dece63","Type":"ContainerStarted","Data":"bfbfb975698b61338c1147253e31c53a65d1b1cb651b84abe755655ff9f89254"}
Sep 17 20:49:25.321388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.321343    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal" podStartSLOduration=2.321308614 podStartE2EDuration="2.321308614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:25.321000843 +0000 UTC m=+5.014810173" watchObservedRunningTime="2026-09-17 20:49:25.321308614 +0000 UTC m=+5.015117944"
Sep 17 20:49:25.823826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.823074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:25.823826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.823138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:25.823826 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.823267    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:25.823826 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.823345    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:27.823308498 +0000 UTC m=+7.517117833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:25.823826 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.823722    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:25.823826 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.823770    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret podName:1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:27.823754642 +0000 UTC m=+7.517563965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret") pod "global-pull-secret-syncer-x66tf" (UID: "1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:25.924006 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:25.923427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:25.924006 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.923607    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:25.924006 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.923628    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:25.924006 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.923641    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt68w for pod openshift-network-diagnostics/network-check-target-r5xpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:25.924006 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:25.923693    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w podName:3df1144b-3aa8-4688-aa04-2e16ff789302 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:27.92367694 +0000 UTC m=+7.617486263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt68w" (UniqueName: "kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w") pod "network-check-target-r5xpp" (UID: "3df1144b-3aa8-4688-aa04-2e16ff789302") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:26.327950 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:26.327678    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f37f127c93297148e7dbac392282653" containerID="adac7651f667bbc861319feaba0a8610e21b78bbb7e92a8e4424aa82ab2dd431" exitCode=0
Sep 17 20:49:26.327950 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:26.327866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" event={"ID":"7f37f127c93297148e7dbac392282653","Type":"ContainerDied","Data":"adac7651f667bbc861319feaba0a8610e21b78bbb7e92a8e4424aa82ab2dd431"}
Sep 17 20:49:27.293341 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:27.293292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:27.293526 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.293449    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:27.293894 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:27.293873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:27.294002 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.293982    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:27.294069 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:27.294057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:27.294154 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.294137    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:27.347489 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:27.347452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" event={"ID":"7f37f127c93297148e7dbac392282653","Type":"ContainerStarted","Data":"7d7d27c75cf707e5c7bf7b6f66fade6a81fda98f46304dc69dc59ee6dbf0d78f"}
Sep 17 20:49:27.839551 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:27.839507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:27.839717 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:27.839590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:27.839775 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.839719    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:27.839834 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.839778    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:31.839758861 +0000 UTC m=+11.533568184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:27.840269 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.840012    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:27.840269 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.840065    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret podName:1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:31.84005007 +0000 UTC m=+11.533859394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret") pod "global-pull-secret-syncer-x66tf" (UID: "1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:27.940802 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:27.940189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:27.940802 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.940381    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:27.940802 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.940405    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:27.940802 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.940418    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt68w for pod openshift-network-diagnostics/network-check-target-r5xpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:27.940802 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:27.940474    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w podName:3df1144b-3aa8-4688-aa04-2e16ff789302 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:31.940457314 +0000 UTC m=+11.634266646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt68w" (UniqueName: "kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w") pod "network-check-target-r5xpp" (UID: "3df1144b-3aa8-4688-aa04-2e16ff789302") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:29.293790 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:29.293754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:29.294251 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:29.293885    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:29.294251 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:29.294240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:29.294437 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:29.294355    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:29.294499 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:29.294445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:29.294574 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:29.294503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:31.294048 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:31.294012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:31.294518 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.294178    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:31.294518 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:31.294444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:31.294636 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.294535    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:31.294636 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:31.294444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:31.294636 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.294604    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:31.874040 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:31.873470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:31.874040 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:31.873533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:31.874040 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.873663    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:31.874040 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.873675    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:31.874040 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.873730    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:39.873710976 +0000 UTC m=+19.567520291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:31.874040 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.873756    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret podName:1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:39.873737642 +0000 UTC m=+19.567546956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret") pod "global-pull-secret-syncer-x66tf" (UID: "1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:31.974601 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:31.974565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:31.974780 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.974722    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:31.974780 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.974741    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:31.974780 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.974752    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt68w for pod openshift-network-diagnostics/network-check-target-r5xpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:31.974929 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:31.974807    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w podName:3df1144b-3aa8-4688-aa04-2e16ff789302 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:39.974789008 +0000 UTC m=+19.668598322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt68w" (UniqueName: "kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w") pod "network-check-target-r5xpp" (UID: "3df1144b-3aa8-4688-aa04-2e16ff789302") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:33.293913 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:33.293883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:33.294421 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:33.293996    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:33.294421 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:33.294303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:33.294527 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:33.294420    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:33.294527 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:33.294461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:33.294626 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:33.294548    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:35.294130 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:35.294100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:35.294597 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:35.294100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:35.294597 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:35.294215    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:35.294597 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:35.294100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:35.294597 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:35.294302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:35.294597 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:35.294424    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:37.294006 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:37.293970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:37.294006 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:37.294008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:37.294483 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:37.293970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:37.294483 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:37.294092    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:37.294483 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:37.294161    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:37.294483 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:37.294281    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:39.294065 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:39.294026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:39.294486 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:39.294144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:39.294486 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:39.294149    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:39.294486 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:39.294202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:39.294486 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:39.294338    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:39.294673 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:39.294494    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:39.928047 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:39.928003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:39.928239 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:39.928076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:39.928239 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:39.928166    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:39.928239 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:39.928166    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:39.928239 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:39.928228    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:55.928211181 +0000 UTC m=+35.622020501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:39.928420 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:39.928245    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret podName:1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:55.928236438 +0000 UTC m=+35.622045746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret") pod "global-pull-secret-syncer-x66tf" (UID: "1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:40.029227 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:40.029158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:40.029399 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:40.029287    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:40.029399 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:40.029330    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:40.029399 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:40.029341    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt68w for pod openshift-network-diagnostics/network-check-target-r5xpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:40.029399 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:40.029397    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w podName:3df1144b-3aa8-4688-aa04-2e16ff789302 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.029379229 +0000 UTC m=+35.723188542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt68w" (UniqueName: "kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w") pod "network-check-target-r5xpp" (UID: "3df1144b-3aa8-4688-aa04-2e16ff789302") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:41.293662 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:41.293632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:41.293662 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:41.293671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:41.294107 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:41.293673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:41.294107 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:41.293725    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:41.294107 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:41.293798    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:41.294107 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:41.293862    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:43.293648 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.293478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:43.294023 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.293663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:43.294023 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.293616    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:43.294023 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:43.293733    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:43.294023 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:43.293832    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:43.294023 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:43.293914    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:43.382210 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.381833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xr4c8" event={"ID":"b0f2f00d-88fd-4ecb-adb7-a1cc9836d870","Type":"ContainerStarted","Data":"b79fba3e7cb85273ddb7ad0fb593b33974c5941d280557f6872e2f091a10b5c6"}
Sep 17 20:49:43.383723 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.383691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6tpr" event={"ID":"8dd712c9-6a04-4eeb-abe9-bc6f45dece63","Type":"ContainerStarted","Data":"9dc99a1abd1d237344cdbc567c3130e090ab851e59a662cbe9c3a7a64ce8b6ac"}
Sep 17 20:49:43.385973 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.385944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76468" event={"ID":"3d96d7a2-ce04-4878-9f9f-e8f3ec947d56","Type":"ContainerStarted","Data":"b327b21ce61c6f6de0b9c074d9e524181be937fb7195376799cbdab2c134503e"}
Sep 17 20:49:43.387550 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.387523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerStarted","Data":"8cbae13689a2cdef2323fedbe1978989d398562798a988fe6002cc43b4e91acb"}
Sep 17 20:49:43.390470 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.388863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cspdc" event={"ID":"2193e970-4de6-43f0-ab19-e94574c01546","Type":"ContainerStarted","Data":"d603a1604e813c529eaa74c5a6d3dc3ae4fd3fa50280945be4057e2cac68d2b4"}
Sep 17 20:49:43.392653 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.392630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j" event={"ID":"03f9073e-d15c-4ce7-8a29-149b0fde2791","Type":"ContainerStarted","Data":"f7a5159366b9c67c9cce7cdedcb6fc081d6fcb035271a45c49b4c34e7131bc6f"}
Sep 17 20:49:43.394090 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.394070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-snzhr" event={"ID":"bf9e641a-5778-4acd-aea2-d08a1b8da14a","Type":"ContainerStarted","Data":"7db9fc70c89a81452e4a6253b80c70bf9b5ceab3b76a5fd3f0c953b4b376aa8b"}
Sep 17 20:49:43.395761 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.395746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:49:43.396074 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.396056    2579 generic.go:356] "Generic (PLEG): container finished" podID="336e56a1-cbd3-497b-b784-4890ae07845a" containerID="6dc13a835fba84db975b24fa83a5429585c1baaac1cc845bee0312dfd61a940a" exitCode=1
Sep 17 20:49:43.396133 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.396084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"76fbf353fd3ba664cd39452464ca9b3dc54714f4b2fc2fc687593d49b07dc28e"}
Sep 17 20:49:43.396172 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.396133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerDied","Data":"6dc13a835fba84db975b24fa83a5429585c1baaac1cc845bee0312dfd61a940a"}
Sep 17 20:49:43.396172 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.396166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"9116e9827a37f9f6ac11287e8bda4a613101a862f3f105eca3b58c94603faff9"}
Sep 17 20:49:43.414829 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.414779    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xr4c8" podStartSLOduration=3.670824347 podStartE2EDuration="21.414761703s" totalImagesPullingTime="17.743937356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.414303004 +0000 UTC m=+23.108112362" watchObservedRunningTime="2026-09-17 20:49:43.414761703 +0000 UTC m=+23.108571034"
Sep 17 20:49:43.415281 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.415209    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" podStartSLOduration=20.415198402 podStartE2EDuration="20.415198402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:27.365222874 +0000 UTC m=+7.059032218" watchObservedRunningTime="2026-09-17 20:49:43.415198402 +0000 UTC m=+23.109007736"
Sep 17 20:49:43.455049 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.454900    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-76468" podStartSLOduration=3.6739674940000002 podStartE2EDuration="21.454881939s" totalImagesPullingTime="17.780914445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.432598681 +0000 UTC m=+23.126408011" watchObservedRunningTime="2026-09-17 20:49:43.454881939 +0000 UTC m=+23.148691271"
Sep 17 20:49:43.473070 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.473026    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cspdc" podStartSLOduration=3.7358990199999997 podStartE2EDuration="21.473010764s" totalImagesPullingTime="17.737111744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.472556235 +0000 UTC m=+23.166365583" watchObservedRunningTime="2026-09-17 20:49:43.473010764 +0000 UTC m=+23.166820095"
Sep 17 20:49:43.492390 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.492341    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t6tpr" podStartSLOduration=7.955567385 podStartE2EDuration="21.49230578s" totalImagesPullingTime="13.536738395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.491784938 +0000 UTC m=+23.185594287" watchObservedRunningTime="2026-09-17 20:49:43.49230578 +0000 UTC m=+23.186115111"
Sep 17 20:49:43.527156 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:43.526731    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-snzhr" podStartSLOduration=3.784582816 podStartE2EDuration="21.526708259s" totalImagesPullingTime="17.742125443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.526410706 +0000 UTC m=+23.220220063" watchObservedRunningTime="2026-09-17 20:49:43.526708259 +0000 UTC m=+23.220517591"
Sep 17 20:49:44.393091 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.393068    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 20:49:44.401600 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.401580    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:49:44.402068 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.402039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"f7eba8ef194b77749d7d77a541ca324b7631335f61e15363d9ccc84173852626"}
Sep 17 20:49:44.402182 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.402120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"d5c888c07382e41b4822802dd6fdaa10930327b616ade141d0cf58406dc39e19"}
Sep 17 20:49:44.402182 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.402161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"ae0183915370b731debf80367e7d783896338f283bff3fc058821598aca1ccff"}
Sep 17 20:49:44.403380 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.403359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ss6d8" event={"ID":"0815ce6c-2ceb-4a9a-9c15-abd5cdb36c85","Type":"ContainerStarted","Data":"43a09d7752b661b79f6ee623821220f791338e735b751a8f4fe406984a7c33af"}
Sep 17 20:49:44.406380 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.405382    2579 generic.go:356] "Generic (PLEG): container finished" podID="720792d8-c7ab-4c51-8ce1-7a79a051e9af" containerID="8cbae13689a2cdef2323fedbe1978989d398562798a988fe6002cc43b4e91acb" exitCode=0
Sep 17 20:49:44.406380 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.405456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerDied","Data":"8cbae13689a2cdef2323fedbe1978989d398562798a988fe6002cc43b4e91acb"}
Sep 17 20:49:44.408386 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.408364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j" event={"ID":"03f9073e-d15c-4ce7-8a29-149b0fde2791","Type":"ContainerStarted","Data":"64b42aade78cb5dcca4e70d4fe37cc7847b79af6fa3bec63292c4a0b9c843763"}
Sep 17 20:49:44.451455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:44.451414    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ss6d8" podStartSLOduration=4.713081735 podStartE2EDuration="22.451401249s" totalImagesPullingTime="17.738319514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:44.423606442 +0000 UTC m=+24.117415772" watchObservedRunningTime="2026-09-17 20:49:44.451401249 +0000 UTC m=+24.145210580"
Sep 17 20:49:45.284481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:45.284357    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:49:44.393085575Z","UUID":"606c9584-11ad-461a-8594-0c7a23ea0729","Handler":null,"Name":"","Endpoint":""}
Sep 17 20:49:45.285919 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:45.285884    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 20:49:45.285919 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:45.285916    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 20:49:45.293659 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:45.293631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:45.293767 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:45.293662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:45.293767 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:45.293738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:45.293767 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:45.293753    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:45.293942 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:45.293822    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:45.293996 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:45.293936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:46.417489 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:46.417285    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:49:46.418028 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:46.417999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"a4989b754e99f165e7994e3f09c0a1b12520d39411c19efedc090108dab8ffd9"}
Sep 17 20:49:46.422723 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:46.422693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j" event={"ID":"03f9073e-d15c-4ce7-8a29-149b0fde2791","Type":"ContainerStarted","Data":"bff58e003804c473a4b20b176c963ad12d6e1d99d9399cb8e2126d7447c3231e"}
Sep 17 20:49:46.440946 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:46.440885    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c969j" podStartSLOduration=4.592885944 podStartE2EDuration="24.440872897s" totalImagesPullingTime="19.847986953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:46.44041861 +0000 UTC m=+26.134227963" watchObservedRunningTime="2026-09-17 20:49:46.440872897 +0000 UTC m=+26.134682228"
Sep 17 20:49:47.293693 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:47.293652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:47.293874 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:47.293707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:47.293874 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:47.293799    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:47.293874 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:47.293846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:47.294031 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:47.293940    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:47.294031 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:47.294012    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:47.866243 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:47.866211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:47.866920 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:47.866900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:48.434765 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:48.434189    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:49:48.435277 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:48.434840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"0d4928bf722667182a685d1d683814a383dadf951f29c79fbb52df90fe67e410"}
Sep 17 20:49:48.435611 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:48.435464    2579 scope.go:117] "RemoveContainer" containerID="6dc13a835fba84db975b24fa83a5429585c1baaac1cc845bee0312dfd61a940a"
Sep 17 20:49:48.435611 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:48.435518    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:48.436998 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:48.436935    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t6tpr"
Sep 17 20:49:49.294028 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.293982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:49.294659 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.293982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:49.294659 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:49.294096    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:49.294659 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.294008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:49.294659 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:49.294168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:49.294659 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:49.294256    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:49.440898 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.440871    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:49:49.441310 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.441285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" event={"ID":"336e56a1-cbd3-497b-b784-4890ae07845a","Type":"ContainerStarted","Data":"e72e1482bd967ab15a853b4b3f1cac40642a3c09b416e0ab07e71ecd54776a5f"}
Sep 17 20:49:49.441509 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.441485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:49.441693 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.441574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:49.441693 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.441628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:49.443625 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.443596    2579 generic.go:356] "Generic (PLEG): container finished" podID="720792d8-c7ab-4c51-8ce1-7a79a051e9af" containerID="b8a0e965c613b415fe1138465a128c7944f669a8e86d1308134cfdb9977c1953" exitCode=0
Sep 17 20:49:49.443767 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.443631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerDied","Data":"b8a0e965c613b415fe1138465a128c7944f669a8e86d1308134cfdb9977c1953"}
Sep 17 20:49:49.457763 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.457745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:49.457861 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.457850    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:49:49.472310 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:49.472274    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b" podStartSLOduration=9.707439766 podStartE2EDuration="27.472260311s" totalImagesPullingTime="17.764820545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:49.471056629 +0000 UTC m=+29.164865981" watchObservedRunningTime="2026-09-17 20:49:49.472260311 +0000 UTC m=+29.166069641"
Sep 17 20:49:50.304524 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.304337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x66tf"]
Sep 17 20:49:50.304913 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.304620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:50.304913 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:50.304714    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:50.304996 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.304983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r5xpp"]
Sep 17 20:49:50.305072 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.305061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:50.305145 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:50.305129    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:50.306951 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.306928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sn8wn"]
Sep 17 20:49:50.307095 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.307034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:50.307158 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:50.307137    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:50.447799 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.447768    2579 generic.go:356] "Generic (PLEG): container finished" podID="720792d8-c7ab-4c51-8ce1-7a79a051e9af" containerID="1978af589c30e08a7e3691bcafc961067ecae635cb185bfc4ccae6e7673ec199" exitCode=0
Sep 17 20:49:50.447948 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:50.447853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerDied","Data":"1978af589c30e08a7e3691bcafc961067ecae635cb185bfc4ccae6e7673ec199"}
Sep 17 20:49:51.452176 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:51.452148    2579 generic.go:356] "Generic (PLEG): container finished" podID="720792d8-c7ab-4c51-8ce1-7a79a051e9af" containerID="7ddd606c6cce44d9fe83548ca16472d53410c83ed66f7c96c82958989032d7ca" exitCode=0
Sep 17 20:49:51.452553 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:51.452237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerDied","Data":"7ddd606c6cce44d9fe83548ca16472d53410c83ed66f7c96c82958989032d7ca"}
Sep 17 20:49:52.295388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:52.295301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:52.295542 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:52.295404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:52.295542 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:52.295449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:52.295542 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:52.295526    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:52.295832 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:52.295802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:52.296024 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:52.295850    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:54.293777 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:54.293688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:54.293777 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:54.293720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:54.293777 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:54.293724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:54.294628 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:54.293816    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x66tf" podUID="1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4"
Sep 17 20:49:54.294628 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:54.293915    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn8wn" podUID="410d7742-1dd0-4ca9-abcf-3b999c572cbf"
Sep 17 20:49:54.294628 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:54.294036    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5xpp" podUID="3df1144b-3aa8-4688-aa04-2e16ff789302"
Sep 17 20:49:55.908236 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.908207    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeReady"
Sep 17 20:49:55.908697 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.908372    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 20:49:55.942652 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.942630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:55.942741 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.942670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:55.942779 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:55.942756    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:55.942779 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:55.942760    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:55.942841 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:55.942803    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:50:27.94279012 +0000 UTC m=+67.636599429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:55.942841 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:55.942816    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret podName:1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:27.942810329 +0000 UTC m=+67.636619638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret") pod "global-pull-secret-syncer-x66tf" (UID: "1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:55.957202 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.957176    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8hwnb"]
Sep 17 20:49:55.985840 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.985786    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dtv9r"]
Sep 17 20:49:55.985949 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.985916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:55.996088 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.996071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 20:49:55.996370 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.996355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gxtkj\""
Sep 17 20:49:55.997373 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.997358    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 20:49:55.997452 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:55.997418    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 20:49:56.001172 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.001152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.001172 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.001165    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hwnb"]
Sep 17 20:49:56.001333 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.001203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dtv9r"]
Sep 17 20:49:56.005775 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.005757    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 20:49:56.007105 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.007089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6qzc\""
Sep 17 20:49:56.007105 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.007097    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 20:49:56.043366 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.043313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598dc\" (UniqueName: \"kubernetes.io/projected/3a96b3a8-897d-4814-af96-b36564baba2d-kube-api-access-598dc\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.043479 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.043400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdftd\" (UniqueName: \"kubernetes.io/projected/030b8144-24d5-47f2-a965-5c9fe63f7882-kube-api-access-cdftd\") pod \"ingress-canary-8hwnb\" (UID: \"030b8144-24d5-47f2-a965-5c9fe63f7882\") " pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:56.043479 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.043428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.043479 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.043460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b3a8-897d-4814-af96-b36564baba2d-tmp-dir\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.043573 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.043528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96b3a8-897d-4814-af96-b36564baba2d-config-volume\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.043573 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.043555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/030b8144-24d5-47f2-a965-5c9fe63f7882-cert\") pod \"ingress-canary-8hwnb\" (UID: \"030b8144-24d5-47f2-a965-5c9fe63f7882\") " pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:56.043651 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.043633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:56.043760 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.043747    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:56.043805 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.043764    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:56.043805 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.043773    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt68w for pod openshift-network-diagnostics/network-check-target-r5xpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:56.043868 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.043823    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w podName:3df1144b-3aa8-4688-aa04-2e16ff789302 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:28.043809472 +0000 UTC m=+67.737618798 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt68w" (UniqueName: "kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w") pod "network-check-target-r5xpp" (UID: "3df1144b-3aa8-4688-aa04-2e16ff789302") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:56.144148 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598dc\" (UniqueName: \"kubernetes.io/projected/3a96b3a8-897d-4814-af96-b36564baba2d-kube-api-access-598dc\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.144252 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdftd\" (UniqueName: \"kubernetes.io/projected/030b8144-24d5-47f2-a965-5c9fe63f7882-kube-api-access-cdftd\") pod \"ingress-canary-8hwnb\" (UID: \"030b8144-24d5-47f2-a965-5c9fe63f7882\") " pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:56.144252 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.144352 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.144259    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:56.144352 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b3a8-897d-4814-af96-b36564baba2d-tmp-dir\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.144352 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.144309    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls podName:3a96b3a8-897d-4814-af96-b36564baba2d nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.644292327 +0000 UTC m=+36.338101655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls") pod "dns-default-dtv9r" (UID: "3a96b3a8-897d-4814-af96-b36564baba2d") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:56.144498 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96b3a8-897d-4814-af96-b36564baba2d-config-volume\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.144498 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/030b8144-24d5-47f2-a965-5c9fe63f7882-cert\") pod \"ingress-canary-8hwnb\" (UID: \"030b8144-24d5-47f2-a965-5c9fe63f7882\") " pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:56.144586 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b3a8-897d-4814-af96-b36564baba2d-tmp-dir\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.144855 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.144835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96b3a8-897d-4814-af96-b36564baba2d-config-volume\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.148279 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.148263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/030b8144-24d5-47f2-a965-5c9fe63f7882-cert\") pod \"ingress-canary-8hwnb\" (UID: \"030b8144-24d5-47f2-a965-5c9fe63f7882\") " pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:56.159515 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.159368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdftd\" (UniqueName: \"kubernetes.io/projected/030b8144-24d5-47f2-a965-5c9fe63f7882-kube-api-access-cdftd\") pod \"ingress-canary-8hwnb\" (UID: \"030b8144-24d5-47f2-a965-5c9fe63f7882\") " pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:56.159816 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.159798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-598dc\" (UniqueName: \"kubernetes.io/projected/3a96b3a8-897d-4814-af96-b36564baba2d-kube-api-access-598dc\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.294126 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.294054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hwnb"
Sep 17 20:49:56.294250 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.294181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:49:56.294286 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.294186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:49:56.294331 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.294299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:49:56.302483 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.302464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:49:56.303576 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.303563    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:49:56.303730 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.303667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-775ms\""
Sep 17 20:49:56.303730 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.303675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:49:56.303876 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.303864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j729c\""
Sep 17 20:49:56.304860 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.304842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:49:56.467247 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.467214    2579 generic.go:356] "Generic (PLEG): container finished" podID="720792d8-c7ab-4c51-8ce1-7a79a051e9af" containerID="5f389104ca60cf4592d161050f9b343fa618e0b060d9da77a9333f3dc6ee1529" exitCode=0
Sep 17 20:49:56.467454 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.467270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerDied","Data":"5f389104ca60cf4592d161050f9b343fa618e0b060d9da77a9333f3dc6ee1529"}
Sep 17 20:49:56.642210 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.642187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hwnb"]
Sep 17 20:49:56.644718 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:49:56.644695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030b8144_24d5_47f2_a965_5c9fe63f7882.slice/crio-1a7e0c77cf21dd3c5571a6facc5d54b71950eef793d64d12e6e08bbf94c92d6c WatchSource:0}: Error finding container 1a7e0c77cf21dd3c5571a6facc5d54b71950eef793d64d12e6e08bbf94c92d6c: Status 404 returned error can't find the container with id 1a7e0c77cf21dd3c5571a6facc5d54b71950eef793d64d12e6e08bbf94c92d6c
Sep 17 20:49:56.648123 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:56.648102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:56.648271 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.648235    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:56.648342 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:56.648312    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls podName:3a96b3a8-897d-4814-af96-b36564baba2d nodeName:}" failed. No retries permitted until 2026-09-17 20:49:57.648296824 +0000 UTC m=+37.342106133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls") pod "dns-default-dtv9r" (UID: "3a96b3a8-897d-4814-af96-b36564baba2d") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:57.471863 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:57.471824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hwnb" event={"ID":"030b8144-24d5-47f2-a965-5c9fe63f7882","Type":"ContainerStarted","Data":"1a7e0c77cf21dd3c5571a6facc5d54b71950eef793d64d12e6e08bbf94c92d6c"}
Sep 17 20:49:57.475604 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:57.475571    2579 generic.go:356] "Generic (PLEG): container finished" podID="720792d8-c7ab-4c51-8ce1-7a79a051e9af" containerID="52664e3bc57965129987bd1ba537f91fea918d5723708d268ef56c6d533447d6" exitCode=0
Sep 17 20:49:57.475716 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:57.475617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerDied","Data":"52664e3bc57965129987bd1ba537f91fea918d5723708d268ef56c6d533447d6"}
Sep 17 20:49:57.655431 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:57.655401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:57.655578 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:57.655557    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:57.655647 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:57.655634    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls podName:3a96b3a8-897d-4814-af96-b36564baba2d nodeName:}" failed. No retries permitted until 2026-09-17 20:49:59.655613875 +0000 UTC m=+39.349423188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls") pod "dns-default-dtv9r" (UID: "3a96b3a8-897d-4814-af96-b36564baba2d") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:58.482291 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:58.482248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sftt" event={"ID":"720792d8-c7ab-4c51-8ce1-7a79a051e9af","Type":"ContainerStarted","Data":"de14e83d22cfb3d9310c224187b024d830dff7cbdec35b53c76ca16e45bf0a23"}
Sep 17 20:49:58.525602 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:58.525531    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4sftt" podStartSLOduration=9.18905061 podStartE2EDuration="36.525513441s" totalImagesPullingTime="27.336462831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:58.523636027 +0000 UTC m=+38.217445388" watchObservedRunningTime="2026-09-17 20:49:58.525513441 +0000 UTC m=+38.219322800"
Sep 17 20:49:59.485900 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:59.485813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hwnb" event={"ID":"030b8144-24d5-47f2-a965-5c9fe63f7882","Type":"ContainerStarted","Data":"81d7d107c06eb88692ad138a055fb0b115de8175b001566b4ac3c21c429b7014"}
Sep 17 20:49:59.508896 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:59.508836    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8hwnb" podStartSLOduration=2.063686053 podStartE2EDuration="4.508815141s" totalImagesPullingTime="2.445129088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:59.507935213 +0000 UTC m=+39.201744555" watchObservedRunningTime="2026-09-17 20:49:59.508815141 +0000 UTC m=+39.202624473"
Sep 17 20:49:59.670303 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:49:59.670259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:49:59.670509 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:59.670419    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:59.670509 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:49:59.670491    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls podName:3a96b3a8-897d-4814-af96-b36564baba2d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:03.670474226 +0000 UTC m=+43.364283549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls") pod "dns-default-dtv9r" (UID: "3a96b3a8-897d-4814-af96-b36564baba2d") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:03.698884 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:03.698848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:50:03.699488 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:03.698976    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:03.699488 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:03.699034    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls podName:3a96b3a8-897d-4814-af96-b36564baba2d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:11.699019016 +0000 UTC m=+51.392828337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls") pod "dns-default-dtv9r" (UID: "3a96b3a8-897d-4814-af96-b36564baba2d") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:07.126305 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.126265    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"]
Sep 17 20:50:07.166012 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.165977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"]
Sep 17 20:50:07.166173 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.166087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"
Sep 17 20:50:07.169468 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.169447    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 20:50:07.169735 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.169714    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mlzm4\""
Sep 17 20:50:07.169787 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.169771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:50:07.219223 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.219194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vl5\" (UniqueName: \"kubernetes.io/projected/a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae-kube-api-access-s8vl5\") pod \"volume-data-source-validator-5c4db8bdf9-c9c2q\" (UID: \"a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"
Sep 17 20:50:07.320054 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.320020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vl5\" (UniqueName: \"kubernetes.io/projected/a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae-kube-api-access-s8vl5\") pod \"volume-data-source-validator-5c4db8bdf9-c9c2q\" (UID: \"a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"
Sep 17 20:50:07.347600 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.347564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vl5\" (UniqueName: \"kubernetes.io/projected/a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae-kube-api-access-s8vl5\") pod \"volume-data-source-validator-5c4db8bdf9-c9c2q\" (UID: \"a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"
Sep 17 20:50:07.398723 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.398658    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-rp4v4"]
Sep 17 20:50:07.414702 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.414680    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64768c58f6-clv7j"]
Sep 17 20:50:07.414824 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.414807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.418281 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.418256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 20:50:07.418439 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.418256    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-t2z2c\""
Sep 17 20:50:07.418858 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.418837    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 20:50:07.419346 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.419301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 20:50:07.419479 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.419457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:50:07.423702 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.423683    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 20:50:07.440665 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.440640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-rp4v4"]
Sep 17 20:50:07.440665 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.440670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64768c58f6-clv7j"]
Sep 17 20:50:07.440832 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.440767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.443230 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.443207    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p2pj8\""
Sep 17 20:50:07.443366 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.443232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 20:50:07.443366 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.443247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 20:50:07.443366 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.443355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 20:50:07.449291 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.449268    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 20:50:07.475552 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.475525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"
Sep 17 20:50:07.521265 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-trusted-ca\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.521265 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-serving-cert\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.521498 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-image-registry-private-configuration\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.521498 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-certificates\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.521498 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vjdj\" (UniqueName: \"kubernetes.io/projected/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-kube-api-access-4vjdj\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.521640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-config\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.521640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-installation-pull-secrets\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.521640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-trusted-ca\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.521640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-bound-sa-token\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.521769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-ca-trust-extracted\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.521769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cnn\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-kube-api-access-p4cnn\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.521769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.521724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.622774 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.622742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-certificates\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.622977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.622802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vjdj\" (UniqueName: \"kubernetes.io/projected/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-kube-api-access-4vjdj\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.622977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.622833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-config\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.622977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.622850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-installation-pull-secrets\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.622977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.622873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-trusted-ca\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.622977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.622888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-bound-sa-token\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.623200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-ca-trust-extracted\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.623200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cnn\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-kube-api-access-p4cnn\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.623200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.623200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-trusted-ca\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.623200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-serving-cert\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.623546 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-image-registry-private-configuration\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.623546 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:07.623276    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:07.623546 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:07.623296    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64768c58f6-clv7j: secret "image-registry-tls" not found
Sep 17 20:50:07.623546 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:07.623385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls podName:ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:08.123366014 +0000 UTC m=+47.817175326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls") pod "image-registry-64768c58f6-clv7j" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d") : secret "image-registry-tls" not found
Sep 17 20:50:07.623713 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-config\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.624000 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.623979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-trusted-ca\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.625598 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.625581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-serving-cert\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.631671 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.631646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vjdj\" (UniqueName: \"kubernetes.io/projected/b7c8009c-f90e-4f56-943d-9e0b8e3414a8-kube-api-access-4vjdj\") pod \"console-operator-fd4bb46c-rp4v4\" (UID: \"b7c8009c-f90e-4f56-943d-9e0b8e3414a8\") " pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.635514 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.635488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-ca-trust-extracted\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.635617 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.635587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-certificates\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.636149 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.636128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-trusted-ca\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.637392 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.637373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-installation-pull-secrets\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.637696 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.637678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cnn\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-kube-api-access-p4cnn\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.637863 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.637848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-bound-sa-token\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.637909 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.637893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-image-registry-private-configuration\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:07.725769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.725684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:07.764157 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:07.764064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q"]
Sep 17 20:50:07.766710 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:07.766682    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fb3dc3_f473_4fff_b4eb_28a2ce8a17ae.slice/crio-1ae006f232abcca2481181ed0368af49c7a5cfab27ee0a15e6f4eaf3bf95410c WatchSource:0}: Error finding container 1ae006f232abcca2481181ed0368af49c7a5cfab27ee0a15e6f4eaf3bf95410c: Status 404 returned error can't find the container with id 1ae006f232abcca2481181ed0368af49c7a5cfab27ee0a15e6f4eaf3bf95410c
Sep 17 20:50:08.019052 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:08.018982    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-rp4v4"]
Sep 17 20:50:08.022186 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:08.022165    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c8009c_f90e_4f56_943d_9e0b8e3414a8.slice/crio-840b9cefd6ebc1406636912cfc5acc8ec56ef43e7c64073a35da45ef64f43410 WatchSource:0}: Error finding container 840b9cefd6ebc1406636912cfc5acc8ec56ef43e7c64073a35da45ef64f43410: Status 404 returned error can't find the container with id 840b9cefd6ebc1406636912cfc5acc8ec56ef43e7c64073a35da45ef64f43410
Sep 17 20:50:08.127419 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:08.127388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:08.127747 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:08.127498    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:08.127747 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:08.127509    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64768c58f6-clv7j: secret "image-registry-tls" not found
Sep 17 20:50:08.127747 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:08.127554    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls podName:ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:09.127542929 +0000 UTC m=+48.821352238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls") pod "image-registry-64768c58f6-clv7j" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d") : secret "image-registry-tls" not found
Sep 17 20:50:08.508144 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:08.508104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" event={"ID":"b7c8009c-f90e-4f56-943d-9e0b8e3414a8","Type":"ContainerStarted","Data":"840b9cefd6ebc1406636912cfc5acc8ec56ef43e7c64073a35da45ef64f43410"}
Sep 17 20:50:08.509614 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:08.509528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q" event={"ID":"a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae","Type":"ContainerStarted","Data":"1ae006f232abcca2481181ed0368af49c7a5cfab27ee0a15e6f4eaf3bf95410c"}
Sep 17 20:50:09.136168 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:09.136136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:09.136661 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:09.136263    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:09.136661 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:09.136276    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64768c58f6-clv7j: secret "image-registry-tls" not found
Sep 17 20:50:09.136661 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:09.136353    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls podName:ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:11.136334517 +0000 UTC m=+50.830143840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls") pod "image-registry-64768c58f6-clv7j" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d") : secret "image-registry-tls" not found
Sep 17 20:50:09.513034 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:09.513000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q" event={"ID":"a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae","Type":"ContainerStarted","Data":"2ece88c533b05e39492595448b82ac4b91487f675fdd7e3852e3b962e883d10a"}
Sep 17 20:50:09.527920 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:09.527874    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c9c2q" podStartSLOduration=0.966817738 podStartE2EDuration="2.527857986s" totalImagesPullingTime="1.561040248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:09.527135683 +0000 UTC m=+49.220945034" watchObservedRunningTime="2026-09-17 20:50:09.527857986 +0000 UTC m=+49.221667317"
Sep 17 20:50:11.055841 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.055760    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wbwfl"]
Sep 17 20:50:11.069839 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.069815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.072262 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.072242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:50:11.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.072658    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:50:11.075684 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.075102    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:50:11.075684 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.075378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h4vct\""
Sep 17 20:50:11.075684 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.075555    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:50:11.076308 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.076280    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wbwfl"]
Sep 17 20:50:11.152632 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.152597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c680a9a3-ba38-4887-9ece-1eaca21028e2-crio-socket\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.152794 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.152656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c680a9a3-ba38-4887-9ece-1eaca21028e2-data-volume\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.152794 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.152680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:11.152794 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.152696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.152794 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.152713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mn9h\" (UniqueName: \"kubernetes.io/projected/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-api-access-8mn9h\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.152794 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.152734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.152954 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.152807    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:11.152954 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.152829    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64768c58f6-clv7j: secret "image-registry-tls" not found
Sep 17 20:50:11.152954 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.152899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls podName:ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:15.152880145 +0000 UTC m=+54.846689468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls") pod "image-registry-64768c58f6-clv7j" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d") : secret "image-registry-tls" not found
Sep 17 20:50:11.253012 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.252979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c680a9a3-ba38-4887-9ece-1eaca21028e2-crio-socket\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.253135 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.253033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c680a9a3-ba38-4887-9ece-1eaca21028e2-data-volume\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.253135 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.253056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.253135 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.253079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mn9h\" (UniqueName: \"kubernetes.io/projected/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-api-access-8mn9h\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.253135 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.253114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.253365 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.253202    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:11.253365 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.253218    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:50:11.253365 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.253201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c680a9a3-ba38-4887-9ece-1eaca21028e2-crio-socket\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.253365 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.253286    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:11.753266964 +0000 UTC m=+51.447076276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:11.253365 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.253315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c680a9a3-ba38-4887-9ece-1eaca21028e2-data-volume\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.253365 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.253314    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:11.753305059 +0000 UTC m=+51.447114381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : configmap "kube-rbac-proxy" not found
Sep 17 20:50:11.268289 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.268263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mn9h\" (UniqueName: \"kubernetes.io/projected/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-api-access-8mn9h\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.520039 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.520006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/0.log"
Sep 17 20:50:11.520220 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.520049    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7c8009c-f90e-4f56-943d-9e0b8e3414a8" containerID="ae4c979bd93181345a3077dcfa281671546d0c441c65dec943c6079059e9ba6a" exitCode=255
Sep 17 20:50:11.520220 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.520108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" event={"ID":"b7c8009c-f90e-4f56-943d-9e0b8e3414a8","Type":"ContainerDied","Data":"ae4c979bd93181345a3077dcfa281671546d0c441c65dec943c6079059e9ba6a"}
Sep 17 20:50:11.520428 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.520412    2579 scope.go:117] "RemoveContainer" containerID="ae4c979bd93181345a3077dcfa281671546d0c441c65dec943c6079059e9ba6a"
Sep 17 20:50:11.758175 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.758130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.758356 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.758182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:11.758356 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:11.758223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:50:11.758356 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.758267    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:11.758356 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.758345    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:50:11.758553 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.758370    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:11.758553 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.758351    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:12.758315462 +0000 UTC m=+52.452124771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:11.758553 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.758418    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls podName:3a96b3a8-897d-4814-af96-b36564baba2d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:27.75840131 +0000 UTC m=+67.452210626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls") pod "dns-default-dtv9r" (UID: "3a96b3a8-897d-4814-af96-b36564baba2d") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:11.758553 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:11.758445    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:12.758434809 +0000 UTC m=+52.452244119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : configmap "kube-rbac-proxy" not found
Sep 17 20:50:12.523896 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.523872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 20:50:12.524295 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.524247    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/0.log"
Sep 17 20:50:12.524295 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.524276    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7c8009c-f90e-4f56-943d-9e0b8e3414a8" containerID="106e2dd5570fca060cc4c1b599c4d2d75acc653e55f93d090386a65a39f91365" exitCode=255
Sep 17 20:50:12.524436 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.524342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" event={"ID":"b7c8009c-f90e-4f56-943d-9e0b8e3414a8","Type":"ContainerDied","Data":"106e2dd5570fca060cc4c1b599c4d2d75acc653e55f93d090386a65a39f91365"}
Sep 17 20:50:12.524436 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.524389    2579 scope.go:117] "RemoveContainer" containerID="ae4c979bd93181345a3077dcfa281671546d0c441c65dec943c6079059e9ba6a"
Sep 17 20:50:12.524632 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.524613    2579 scope.go:117] "RemoveContainer" containerID="106e2dd5570fca060cc4c1b599c4d2d75acc653e55f93d090386a65a39f91365"
Sep 17 20:50:12.524799 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:12.524779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-rp4v4_openshift-console-operator(b7c8009c-f90e-4f56-943d-9e0b8e3414a8)\"" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" podUID="b7c8009c-f90e-4f56-943d-9e0b8e3414a8"
Sep 17 20:50:12.767575 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.767540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:12.767734 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:12.767582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:12.767734 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:12.767644    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:50:12.767734 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:12.767683    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:12.767734 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:12.767694    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:14.767681886 +0000 UTC m=+54.461491195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : configmap "kube-rbac-proxy" not found
Sep 17 20:50:12.767872 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:12.767740    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:14.767727543 +0000 UTC m=+54.461536855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:13.174493 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:13.174468    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xr4c8_b0f2f00d-88fd-4ecb-adb7-a1cc9836d870/dns-node-resolver/0.log"
Sep 17 20:50:13.528395 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:13.528305    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 20:50:13.528716 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:13.528659    2579 scope.go:117] "RemoveContainer" containerID="106e2dd5570fca060cc4c1b599c4d2d75acc653e55f93d090386a65a39f91365"
Sep 17 20:50:13.528855 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:13.528837    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-rp4v4_openshift-console-operator(b7c8009c-f90e-4f56-943d-9e0b8e3414a8)\"" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" podUID="b7c8009c-f90e-4f56-943d-9e0b8e3414a8"
Sep 17 20:50:14.175027 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:14.175003    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cspdc_2193e970-4de6-43f0-ab19-e94574c01546/node-ca/0.log"
Sep 17 20:50:14.774965 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:14.774935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8hwnb_030b8144-24d5-47f2-a965-5c9fe63f7882/serve-healthcheck-canary/0.log"
Sep 17 20:50:14.781153 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:14.781127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:14.781271 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:14.781173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:14.781271 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:14.781264    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:50:14.781364 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:14.781265    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:14.781364 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:14.781345    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:18.781313258 +0000 UTC m=+58.475122567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : configmap "kube-rbac-proxy" not found
Sep 17 20:50:14.781364 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:14.781361    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:18.781355521 +0000 UTC m=+58.475164830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:15.184803 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:15.184765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:15.184968 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:15.184921    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:15.184968 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:15.184940    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64768c58f6-clv7j: secret "image-registry-tls" not found
Sep 17 20:50:15.185045 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:15.184993    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls podName:ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:23.184977808 +0000 UTC m=+62.878787117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls") pod "image-registry-64768c58f6-clv7j" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d") : secret "image-registry-tls" not found
Sep 17 20:50:17.726749 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:17.726718    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:17.727186 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:17.726775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:17.727186 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:17.727064    2579 scope.go:117] "RemoveContainer" containerID="106e2dd5570fca060cc4c1b599c4d2d75acc653e55f93d090386a65a39f91365"
Sep 17 20:50:17.727254 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:17.727192    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-rp4v4_openshift-console-operator(b7c8009c-f90e-4f56-943d-9e0b8e3414a8)\"" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" podUID="b7c8009c-f90e-4f56-943d-9e0b8e3414a8"
Sep 17 20:50:18.813135 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:18.813096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:18.813135 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:18.813139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:18.813636 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:18.813255    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:18.813636 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:18.813336    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.813303325 +0000 UTC m=+66.507112638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:18.813636 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:18.813258    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:50:18.813636 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:18.813423    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.813408115 +0000 UTC m=+66.507217424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : configmap "kube-rbac-proxy" not found
Sep 17 20:50:19.465543 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.465514    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xx88l"]
Sep 17 20:50:19.469527 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.469513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.472416 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.472390    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 20:50:19.472521 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.472394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kcwwc\""
Sep 17 20:50:19.473173 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.473153    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 20:50:19.473268 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.473204    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 20:50:19.473424 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.473398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 20:50:19.479912 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.479894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xx88l"]
Sep 17 20:50:19.620580 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.620550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d96cdd9-3822-495e-b66d-6056c6942299-signing-cabundle\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.620580 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.620584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt684\" (UniqueName: \"kubernetes.io/projected/4d96cdd9-3822-495e-b66d-6056c6942299-kube-api-access-rt684\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.620786 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.620691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d96cdd9-3822-495e-b66d-6056c6942299-signing-key\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.722158 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.722063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d96cdd9-3822-495e-b66d-6056c6942299-signing-key\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.722158 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.722145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d96cdd9-3822-495e-b66d-6056c6942299-signing-cabundle\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.722158 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.722164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt684\" (UniqueName: \"kubernetes.io/projected/4d96cdd9-3822-495e-b66d-6056c6942299-kube-api-access-rt684\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.722879 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.722813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d96cdd9-3822-495e-b66d-6056c6942299-signing-cabundle\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.724610 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.724582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d96cdd9-3822-495e-b66d-6056c6942299-signing-key\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.731020 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.731003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt684\" (UniqueName: \"kubernetes.io/projected/4d96cdd9-3822-495e-b66d-6056c6942299-kube-api-access-rt684\") pod \"service-ca-55f6fc7b86-xx88l\" (UID: \"4d96cdd9-3822-495e-b66d-6056c6942299\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:19.778106 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:19.778082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l"
Sep 17 20:50:20.051580 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:20.048975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xx88l"]
Sep 17 20:50:20.054599 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:20.054572    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d96cdd9_3822_495e_b66d_6056c6942299.slice/crio-fae7b249bde0649e5f7508fbda9a3a18ec7f54860dddc76ba06468fbebd99c03 WatchSource:0}: Error finding container fae7b249bde0649e5f7508fbda9a3a18ec7f54860dddc76ba06468fbebd99c03: Status 404 returned error can't find the container with id fae7b249bde0649e5f7508fbda9a3a18ec7f54860dddc76ba06468fbebd99c03
Sep 17 20:50:20.546877 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:20.546840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l" event={"ID":"4d96cdd9-3822-495e-b66d-6056c6942299","Type":"ContainerStarted","Data":"fae7b249bde0649e5f7508fbda9a3a18ec7f54860dddc76ba06468fbebd99c03"}
Sep 17 20:50:21.463069 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:21.463039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khg2b"
Sep 17 20:50:22.554404 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:22.554279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l" event={"ID":"4d96cdd9-3822-495e-b66d-6056c6942299","Type":"ContainerStarted","Data":"63ae2df54c94fa297df89e733481535442a67428a1ac62a5c03467f25c4e9258"}
Sep 17 20:50:22.572655 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:22.572613    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-xx88l" podStartSLOduration=1.505837887 podStartE2EDuration="3.572598423s" totalImagesPullingTime="2.066760536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:22.572138316 +0000 UTC m=+62.265947648" watchObservedRunningTime="2026-09-17 20:50:22.572598423 +0000 UTC m=+62.266407753"
Sep 17 20:50:23.251757 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:23.251718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:23.251958 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:23.251853    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:23.251958 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:23.251865    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64768c58f6-clv7j: secret "image-registry-tls" not found
Sep 17 20:50:23.251958 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:23.251915    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls podName:ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:39.251900929 +0000 UTC m=+78.945710250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls") pod "image-registry-64768c58f6-clv7j" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d") : secret "image-registry-tls" not found
Sep 17 20:50:26.882923 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:26.882888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:26.883362 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:26.882933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:26.883362 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:26.883042    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:26.883362 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:26.883117    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls podName:c680a9a3-ba38-4887-9ece-1eaca21028e2 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:42.8830995 +0000 UTC m=+82.576908808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwfl" (UID: "c680a9a3-ba38-4887-9ece-1eaca21028e2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:50:26.883611 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:26.883592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680a9a3-ba38-4887-9ece-1eaca21028e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:27.791331 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.791291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:50:27.793802 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.793771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a96b3a8-897d-4814-af96-b36564baba2d-metrics-tls\") pod \"dns-default-dtv9r\" (UID: \"3a96b3a8-897d-4814-af96-b36564baba2d\") " pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:50:27.812556 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.812532    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6qzc\""
Sep 17 20:50:27.820804 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.820776    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:50:27.993528 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.993499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:50:27.993939 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.993561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:50:27.996240 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.996210    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:27.996240 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:27.996232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:28.004607 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:28.004589    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:50:28.004699 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:28.004643    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs podName:410d7742-1dd0-4ca9-abcf-3b999c572cbf nodeName:}" failed. No retries permitted until 2026-09-17 20:51:32.004628002 +0000 UTC m=+131.698437324 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs") pod "network-metrics-daemon-sn8wn" (UID: "410d7742-1dd0-4ca9-abcf-3b999c572cbf") : secret "metrics-daemon-secret" not found
Sep 17 20:50:28.007055 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.007035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4-original-pull-secret\") pod \"global-pull-secret-syncer-x66tf\" (UID: \"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4\") " pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:50:28.094134 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.094106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:50:28.096673 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.096654    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:28.104750 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.104728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x66tf"
Sep 17 20:50:28.106799 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.106779    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:28.117042 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.117017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt68w\" (UniqueName: \"kubernetes.io/projected/3df1144b-3aa8-4688-aa04-2e16ff789302-kube-api-access-dt68w\") pod \"network-check-target-r5xpp\" (UID: \"3df1144b-3aa8-4688-aa04-2e16ff789302\") " pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:50:28.120453 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.120428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dtv9r"]
Sep 17 20:50:28.124420 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:28.124389    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a96b3a8_897d_4814_af96_b36564baba2d.slice/crio-54327a0b8f579f8e5be45593df9473c507f1a14093626db59289f3a52f95fc24 WatchSource:0}: Error finding container 54327a0b8f579f8e5be45593df9473c507f1a14093626db59289f3a52f95fc24: Status 404 returned error can't find the container with id 54327a0b8f579f8e5be45593df9473c507f1a14093626db59289f3a52f95fc24
Sep 17 20:50:28.393544 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.393470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x66tf"]
Sep 17 20:50:28.396430 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:28.396400    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db2c98f_88ed_40a4_aa18_e2dc3a5bddd4.slice/crio-2e7e71592ebd4cc0b87ed684bc0587017bdfe079959afdd32037cf0322cc0506 WatchSource:0}: Error finding container 2e7e71592ebd4cc0b87ed684bc0587017bdfe079959afdd32037cf0322cc0506: Status 404 returned error can't find the container with id 2e7e71592ebd4cc0b87ed684bc0587017bdfe079959afdd32037cf0322cc0506
Sep 17 20:50:28.416879 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.416853    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j729c\""
Sep 17 20:50:28.424857 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.424838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:50:28.571390 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.571351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dtv9r" event={"ID":"3a96b3a8-897d-4814-af96-b36564baba2d","Type":"ContainerStarted","Data":"54327a0b8f579f8e5be45593df9473c507f1a14093626db59289f3a52f95fc24"}
Sep 17 20:50:28.572480 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.572446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x66tf" event={"ID":"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4","Type":"ContainerStarted","Data":"2e7e71592ebd4cc0b87ed684bc0587017bdfe079959afdd32037cf0322cc0506"}
Sep 17 20:50:28.724387 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:28.724259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r5xpp"]
Sep 17 20:50:28.728625 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:28.728592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df1144b_3aa8_4688_aa04_2e16ff789302.slice/crio-123ec3a8a3430de69da647bee44a8b9172072069d1498a477540faa688577718 WatchSource:0}: Error finding container 123ec3a8a3430de69da647bee44a8b9172072069d1498a477540faa688577718: Status 404 returned error can't find the container with id 123ec3a8a3430de69da647bee44a8b9172072069d1498a477540faa688577718
Sep 17 20:50:29.578167 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:29.578113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r5xpp" event={"ID":"3df1144b-3aa8-4688-aa04-2e16ff789302","Type":"ContainerStarted","Data":"123ec3a8a3430de69da647bee44a8b9172072069d1498a477540faa688577718"}
Sep 17 20:50:30.585201 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:30.585159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dtv9r" event={"ID":"3a96b3a8-897d-4814-af96-b36564baba2d","Type":"ContainerStarted","Data":"59a35a0189b280ae7b0712e5fd5adab11d875174e900d19b570a1857c4794729"}
Sep 17 20:50:30.585680 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:30.585226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dtv9r" event={"ID":"3a96b3a8-897d-4814-af96-b36564baba2d","Type":"ContainerStarted","Data":"6e510196d9fd5f738ab1256ac125fd200eea4d556f42ad4a094ecb51b8a300d0"}
Sep 17 20:50:30.585680 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:30.585514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:50:30.604375 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:30.604279    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dtv9r" podStartSLOduration=33.467290488 podStartE2EDuration="35.604259178s" totalImagesPullingTime="2.13696869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:30.602823037 +0000 UTC m=+70.296632403" watchObservedRunningTime="2026-09-17 20:50:30.604259178 +0000 UTC m=+70.298068510"
Sep 17 20:50:31.294834 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:31.294802    2579 scope.go:117] "RemoveContainer" containerID="106e2dd5570fca060cc4c1b599c4d2d75acc653e55f93d090386a65a39f91365"
Sep 17 20:50:33.600031 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.600001    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 20:50:33.600437 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.600098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" event={"ID":"b7c8009c-f90e-4f56-943d-9e0b8e3414a8","Type":"ContainerStarted","Data":"1ba64229c9373b3ffbb71a4e2774209737dfc6a3b559824fe4f847a6b601fb23"}
Sep 17 20:50:33.600499 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.600484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:33.602645 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.602609    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r5xpp" event={"ID":"3df1144b-3aa8-4688-aa04-2e16ff789302","Type":"ContainerStarted","Data":"15d5e554c122950dc4586328b42c9c7bd7db194b9ee06890b57b13d5a49df4a1"}
Sep 17 20:50:33.602791 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.602656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:50:33.604644 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.604585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x66tf" event={"ID":"1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4","Type":"ContainerStarted","Data":"39d723633300aa61b328ab3a5d0036f2bdb47e4b099db9e575c72f1f019ccaa7"}
Sep 17 20:50:33.618126 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.618084    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4" podStartSLOduration=23.99384689 podStartE2EDuration="26.61806997s" totalImagesPullingTime="2.62422308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:33.616967471 +0000 UTC m=+73.310776803" watchObservedRunningTime="2026-09-17 20:50:33.61806997 +0000 UTC m=+73.311879313"
Sep 17 20:50:33.632617 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.632564    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x66tf" podStartSLOduration=66.570905108 podStartE2EDuration="1m11.632547331s" totalImagesPullingTime="5.061642223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:33.632204103 +0000 UTC m=+73.326013453" watchObservedRunningTime="2026-09-17 20:50:33.632547331 +0000 UTC m=+73.326356663"
Sep 17 20:50:33.649703 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.649590    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r5xpp" podStartSLOduration=66.927723776 podStartE2EDuration="1m11.649568441s" totalImagesPullingTime="4.721844665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:33.648726414 +0000 UTC m=+73.342535747" watchObservedRunningTime="2026-09-17 20:50:33.649568441 +0000 UTC m=+73.343377763"
Sep 17 20:50:33.969209 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:33.969106    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-rp4v4"
Sep 17 20:50:39.279196 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:39.279150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:39.281722 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:39.281700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"image-registry-64768c58f6-clv7j\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") " pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:39.558832 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:39.558753    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p2pj8\""
Sep 17 20:50:39.563908 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:39.563894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:39.850849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:39.850819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64768c58f6-clv7j"]
Sep 17 20:50:39.876718 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:39.876683    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb0440a_2cb0_4e1d_833c_bb3feccc2e9d.slice/crio-665ba352f6713667691ab10b09d1abee31c5b3167008582aa64c8ccf4fe6bef5 WatchSource:0}: Error finding container 665ba352f6713667691ab10b09d1abee31c5b3167008582aa64c8ccf4fe6bef5: Status 404 returned error can't find the container with id 665ba352f6713667691ab10b09d1abee31c5b3167008582aa64c8ccf4fe6bef5
Sep 17 20:50:40.591349 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:40.591301    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dtv9r"
Sep 17 20:50:40.631156 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:40.631123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64768c58f6-clv7j" event={"ID":"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d","Type":"ContainerStarted","Data":"42251411127fc3c1e9882a42abd25201a92979dedebcd50d2eb5c2ae5a2f5b30"}
Sep 17 20:50:40.631335 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:40.631167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64768c58f6-clv7j" event={"ID":"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d","Type":"ContainerStarted","Data":"665ba352f6713667691ab10b09d1abee31c5b3167008582aa64c8ccf4fe6bef5"}
Sep 17 20:50:40.631335 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:40.631251    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:50:40.665920 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:40.665876    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64768c58f6-clv7j" podStartSLOduration=33.665861686 podStartE2EDuration="33.665861686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:40.665861893 +0000 UTC m=+80.359671249" watchObservedRunningTime="2026-09-17 20:50:40.665861686 +0000 UTC m=+80.359671017"
Sep 17 20:50:42.904741 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:42.904692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:42.907106 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:42.907082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680a9a3-ba38-4887-9ece-1eaca21028e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwfl\" (UID: \"c680a9a3-ba38-4887-9ece-1eaca21028e2\") " pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:43.198222 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.198154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h4vct\""
Sep 17 20:50:43.202439 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.202426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wbwfl"
Sep 17 20:50:43.500009 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.499933    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wbwfl"]
Sep 17 20:50:43.509866 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:43.509829    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc680a9a3_ba38_4887_9ece_1eaca21028e2.slice/crio-601e57593978125a9a77fe7dc2d1aaa42052820a83d2ab0622bde163b79b3a22 WatchSource:0}: Error finding container 601e57593978125a9a77fe7dc2d1aaa42052820a83d2ab0622bde163b79b3a22: Status 404 returned error can't find the container with id 601e57593978125a9a77fe7dc2d1aaa42052820a83d2ab0622bde163b79b3a22
Sep 17 20:50:43.625180 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.625130    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"]
Sep 17 20:50:43.628760 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.628743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.634679 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.634601    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 20:50:43.635381 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.635363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 20:50:43.635504 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.635457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 20:50:43.635723 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.635709    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 20:50:43.642917 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.642898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwfl" event={"ID":"c680a9a3-ba38-4887-9ece-1eaca21028e2","Type":"ContainerStarted","Data":"57e27f75b1d6102e681fa9ef72428d5c93e27037f21c742123ecbb4e0e7b1034"}
Sep 17 20:50:43.642986 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.642942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwfl" event={"ID":"c680a9a3-ba38-4887-9ece-1eaca21028e2","Type":"ContainerStarted","Data":"601e57593978125a9a77fe7dc2d1aaa42052820a83d2ab0622bde163b79b3a22"}
Sep 17 20:50:43.696913 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.696884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"]
Sep 17 20:50:43.711429 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.711405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/406411f2-11a9-4ff9-b218-453040c403ea-tmp\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.711548 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.711445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbqmd\" (UniqueName: \"kubernetes.io/projected/406411f2-11a9-4ff9-b218-453040c403ea-kube-api-access-vbqmd\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.711548 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.711513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/406411f2-11a9-4ff9-b218-453040c403ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.812869 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.812786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/406411f2-11a9-4ff9-b218-453040c403ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.812869 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.812832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/406411f2-11a9-4ff9-b218-453040c403ea-tmp\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.813052 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.812873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbqmd\" (UniqueName: \"kubernetes.io/projected/406411f2-11a9-4ff9-b218-453040c403ea-kube-api-access-vbqmd\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.813249 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.813230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/406411f2-11a9-4ff9-b218-453040c403ea-tmp\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.815869 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.815848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/406411f2-11a9-4ff9-b218-453040c403ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.889730 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.889698    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-h5nzh"]
Sep 17 20:50:43.893148 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.893133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-h5nzh"
Sep 17 20:50:43.904308 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.904285    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64768c58f6-clv7j"]
Sep 17 20:50:43.914103 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.914082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmnf\" (UniqueName: \"kubernetes.io/projected/32dd64bc-fb24-498d-a947-deb5e66032b9-kube-api-access-xxmnf\") pod \"downloads-6594f4c75-h5nzh\" (UID: \"32dd64bc-fb24-498d-a947-deb5e66032b9\") " pod="openshift-console/downloads-6594f4c75-h5nzh"
Sep 17 20:50:43.937191 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.937108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h7xmh\""
Sep 17 20:50:43.939162 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.939142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbqmd\" (UniqueName: \"kubernetes.io/projected/406411f2-11a9-4ff9-b218-453040c403ea-kube-api-access-vbqmd\") pod \"klusterlet-addon-workmgr-649659f9dc-4btxk\" (UID: \"406411f2-11a9-4ff9-b218-453040c403ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:43.944368 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.944351    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 20:50:43.948386 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.948370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 20:50:43.964850 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:43.964830    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-h5nzh"]
Sep 17 20:50:44.014585 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.014558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmnf\" (UniqueName: \"kubernetes.io/projected/32dd64bc-fb24-498d-a947-deb5e66032b9-kube-api-access-xxmnf\") pod \"downloads-6594f4c75-h5nzh\" (UID: \"32dd64bc-fb24-498d-a947-deb5e66032b9\") " pod="openshift-console/downloads-6594f4c75-h5nzh"
Sep 17 20:50:44.093846 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.093815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmnf\" (UniqueName: \"kubernetes.io/projected/32dd64bc-fb24-498d-a947-deb5e66032b9-kube-api-access-xxmnf\") pod \"downloads-6594f4c75-h5nzh\" (UID: \"32dd64bc-fb24-498d-a947-deb5e66032b9\") " pod="openshift-console/downloads-6594f4c75-h5nzh"
Sep 17 20:50:44.201222 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.201193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-h5nzh"
Sep 17 20:50:44.205501 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.205476    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6445b7fc9f-gj4gh"]
Sep 17 20:50:44.208353 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.208334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.232611 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.232582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6445b7fc9f-gj4gh"]
Sep 17 20:50:44.237654 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.237631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:44.317458 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adf57d26-bb0b-4a67-8208-8e375949907c-image-registry-private-configuration\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.317586 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adf57d26-bb0b-4a67-8208-8e375949907c-registry-certificates\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.317586 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adf57d26-bb0b-4a67-8208-8e375949907c-trusted-ca\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.317586 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adf57d26-bb0b-4a67-8208-8e375949907c-ca-trust-extracted\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.317766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-registry-tls\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.317766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsdb\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-kube-api-access-dtsdb\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.317766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adf57d26-bb0b-4a67-8208-8e375949907c-installation-pull-secrets\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.317766 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.317683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-bound-sa-token\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.418839 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.418759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adf57d26-bb0b-4a67-8208-8e375949907c-ca-trust-extracted\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.418839 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.418794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-registry-tls\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.418839 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.418814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsdb\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-kube-api-access-dtsdb\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.419058 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.418948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adf57d26-bb0b-4a67-8208-8e375949907c-installation-pull-secrets\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.419058 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.418985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-bound-sa-token\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.419058 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.419038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adf57d26-bb0b-4a67-8208-8e375949907c-image-registry-private-configuration\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.419195 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.419082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adf57d26-bb0b-4a67-8208-8e375949907c-registry-certificates\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.419195 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.419136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adf57d26-bb0b-4a67-8208-8e375949907c-trusted-ca\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.419357 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.419208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adf57d26-bb0b-4a67-8208-8e375949907c-ca-trust-extracted\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.420171 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.420130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adf57d26-bb0b-4a67-8208-8e375949907c-registry-certificates\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.420294 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.420258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adf57d26-bb0b-4a67-8208-8e375949907c-trusted-ca\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.421634 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.421596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-registry-tls\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.421634 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.421617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adf57d26-bb0b-4a67-8208-8e375949907c-installation-pull-secrets\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.421810 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.421742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adf57d26-bb0b-4a67-8208-8e375949907c-image-registry-private-configuration\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.431476 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.431452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsdb\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-kube-api-access-dtsdb\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.431826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.431810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adf57d26-bb0b-4a67-8208-8e375949907c-bound-sa-token\") pod \"image-registry-6445b7fc9f-gj4gh\" (UID: \"adf57d26-bb0b-4a67-8208-8e375949907c\") " pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.498621 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.498599    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-h5nzh"]
Sep 17 20:50:44.500861 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:44.500834    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32dd64bc_fb24_498d_a947_deb5e66032b9.slice/crio-c3f008d7208bcd0ed114a2bb5b2fc85273805b057dc193d1c2d56728a4227099 WatchSource:0}: Error finding container c3f008d7208bcd0ed114a2bb5b2fc85273805b057dc193d1c2d56728a4227099: Status 404 returned error can't find the container with id c3f008d7208bcd0ed114a2bb5b2fc85273805b057dc193d1c2d56728a4227099
Sep 17 20:50:44.527740 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.527721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:44.531763 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.531691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"]
Sep 17 20:50:44.536569 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:44.536549    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406411f2_11a9_4ff9_b218_453040c403ea.slice/crio-ed90e3a55bbede917df2389a4e3d3fa904c4ff3521c2ed2fb8ae34d7b9048197 WatchSource:0}: Error finding container ed90e3a55bbede917df2389a4e3d3fa904c4ff3521c2ed2fb8ae34d7b9048197: Status 404 returned error can't find the container with id ed90e3a55bbede917df2389a4e3d3fa904c4ff3521c2ed2fb8ae34d7b9048197
Sep 17 20:50:44.647005 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.646967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-h5nzh" event={"ID":"32dd64bc-fb24-498d-a947-deb5e66032b9","Type":"ContainerStarted","Data":"c3f008d7208bcd0ed114a2bb5b2fc85273805b057dc193d1c2d56728a4227099"}
Sep 17 20:50:44.648608 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.648578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwfl" event={"ID":"c680a9a3-ba38-4887-9ece-1eaca21028e2","Type":"ContainerStarted","Data":"6d474ad0733404f7af55afaac242c4f7c0464288e719f67998124fbf58ab38d6"}
Sep 17 20:50:44.649603 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.649587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk" event={"ID":"406411f2-11a9-4ff9-b218-453040c403ea","Type":"ContainerStarted","Data":"ed90e3a55bbede917df2389a4e3d3fa904c4ff3521c2ed2fb8ae34d7b9048197"}
Sep 17 20:50:44.828597 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:44.828573    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6445b7fc9f-gj4gh"]
Sep 17 20:50:44.830615 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:44.830591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf57d26_bb0b_4a67_8208_8e375949907c.slice/crio-369a51c75aacead3aee1c90143dabb2e8efce46b2abc6d0afa7a9f6be223dc50 WatchSource:0}: Error finding container 369a51c75aacead3aee1c90143dabb2e8efce46b2abc6d0afa7a9f6be223dc50: Status 404 returned error can't find the container with id 369a51c75aacead3aee1c90143dabb2e8efce46b2abc6d0afa7a9f6be223dc50
Sep 17 20:50:45.657082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:45.656616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh" event={"ID":"adf57d26-bb0b-4a67-8208-8e375949907c","Type":"ContainerStarted","Data":"cf38c7a0eed676860b9f68e8c0e57c9ebaf51e28e6e42add03156e087a75e22c"}
Sep 17 20:50:45.657082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:45.656676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh" event={"ID":"adf57d26-bb0b-4a67-8208-8e375949907c","Type":"ContainerStarted","Data":"369a51c75aacead3aee1c90143dabb2e8efce46b2abc6d0afa7a9f6be223dc50"}
Sep 17 20:50:45.657082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:45.656750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:50:45.689035 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:45.688987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh" podStartSLOduration=1.688970765 podStartE2EDuration="1.688970765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:45.688070471 +0000 UTC m=+85.381879821" watchObservedRunningTime="2026-09-17 20:50:45.688970765 +0000 UTC m=+85.382780096"
Sep 17 20:50:46.666044 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:46.665965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwfl" event={"ID":"c680a9a3-ba38-4887-9ece-1eaca21028e2","Type":"ContainerStarted","Data":"c67c2c60d9660bfb39a5ce07ba9a0d8b4fd7be957948958e8e4341376867ad4f"}
Sep 17 20:50:46.699270 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:46.699184    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wbwfl" podStartSLOduration=33.619975783 podStartE2EDuration="35.699162349s" totalImagesPullingTime="2.079186566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:50:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:46.69706825 +0000 UTC m=+86.390877608" watchObservedRunningTime="2026-09-17 20:50:46.699162349 +0000 UTC m=+86.392971680"
Sep 17 20:50:49.681413 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:49.681375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk" event={"ID":"406411f2-11a9-4ff9-b218-453040c403ea","Type":"ContainerStarted","Data":"02daa003b7009ba61ebe1e62d53d5f443362e661dccbb1adb992a1ec555ab49d"}
Sep 17 20:50:49.682005 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:49.681587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:49.683481 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:49.683458    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk"
Sep 17 20:50:49.699768 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:49.699715    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649659f9dc-4btxk" podStartSLOduration=2.042040552 podStartE2EDuration="6.699697988s" totalImagesPullingTime="4.657657436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:49.69899844 +0000 UTC m=+89.392807807" watchObservedRunningTime="2026-09-17 20:50:49.699697988 +0000 UTC m=+89.393507320"
Sep 17 20:50:52.238764 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.238731    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"]
Sep 17 20:50:52.243928 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.243906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.247782 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.247761    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 20:50:52.247936 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.247917    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 20:50:52.248022 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.248005    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 20:50:52.248383 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.248368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 20:50:52.248995 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.248980    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 20:50:52.249082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.249026    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mvhm7\""
Sep 17 20:50:52.261905 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.261881    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"]
Sep 17 20:50:52.284601 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.284502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.284601 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.284579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dae0b2e-b717-48f6-8042-81fd7c11054f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.284765 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.284661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.284765 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.284713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46frg\" (UniqueName: \"kubernetes.io/projected/5dae0b2e-b717-48f6-8042-81fd7c11054f-kube-api-access-46frg\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.385559 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.385521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dae0b2e-b717-48f6-8042-81fd7c11054f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.385751 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.385588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.385751 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.385609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46frg\" (UniqueName: \"kubernetes.io/projected/5dae0b2e-b717-48f6-8042-81fd7c11054f-kube-api-access-46frg\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.385751 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.385671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.385751 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:52.385745    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 20:50:52.385960 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:52.385815    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-tls podName:5dae0b2e-b717-48f6-8042-81fd7c11054f nodeName:}" failed. No retries permitted until 2026-09-17 20:50:52.885795871 +0000 UTC m=+92.579605206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-t9fln" (UID: "5dae0b2e-b717-48f6-8042-81fd7c11054f") : secret "prometheus-operator-tls" not found
Sep 17 20:50:52.386338 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.386287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dae0b2e-b717-48f6-8042-81fd7c11054f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.388640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.388616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.398505 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.398481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46frg\" (UniqueName: \"kubernetes.io/projected/5dae0b2e-b717-48f6-8042-81fd7c11054f-kube-api-access-46frg\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.889283 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.889246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:52.892078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:52.892053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dae0b2e-b717-48f6-8042-81fd7c11054f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-t9fln\" (UID: \"5dae0b2e-b717-48f6-8042-81fd7c11054f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:53.153367 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:53.153275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"
Sep 17 20:50:53.435089 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:53.435007    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln"]
Sep 17 20:50:53.438667 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:50:53.438638    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dae0b2e_b717_48f6_8042_81fd7c11054f.slice/crio-e05b7471afb37e208e2fa0fb5199e820f891ec3097fbe513d4e877d9411d7fc3 WatchSource:0}: Error finding container e05b7471afb37e208e2fa0fb5199e820f891ec3097fbe513d4e877d9411d7fc3: Status 404 returned error can't find the container with id e05b7471afb37e208e2fa0fb5199e820f891ec3097fbe513d4e877d9411d7fc3
Sep 17 20:50:53.698894 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:53.698807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln" event={"ID":"5dae0b2e-b717-48f6-8042-81fd7c11054f","Type":"ContainerStarted","Data":"e05b7471afb37e208e2fa0fb5199e820f891ec3097fbe513d4e877d9411d7fc3"}
Sep 17 20:50:55.708832 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:55.708798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln" event={"ID":"5dae0b2e-b717-48f6-8042-81fd7c11054f","Type":"ContainerStarted","Data":"7cd21870860925911887ef27bdca0a58208dd9032edff866d8760a4f5b560a07"}
Sep 17 20:50:55.709270 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:55.708860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln" event={"ID":"5dae0b2e-b717-48f6-8042-81fd7c11054f","Type":"ContainerStarted","Data":"0a422165749a02c7680f40ff219bbd05a9f9486aa54aaac7d190d582797515a4"}
Sep 17 20:50:55.727367 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:55.727294    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t9fln" podStartSLOduration=2.157132271 podStartE2EDuration="3.72728005s" totalImagesPullingTime="1.570147779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:55.726456188 +0000 UTC m=+95.420265552" watchObservedRunningTime="2026-09-17 20:50:55.72728005 +0000 UTC m=+95.421089381"
Sep 17 20:50:57.629212 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.629107    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vl9j4"]
Sep 17 20:50:57.638304 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.636597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.642166 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.641825    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 20:50:57.642166 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.641829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vfzf5\""
Sep 17 20:50:57.642166 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.642033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 20:50:57.643060 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.642834    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 20:50:57.733583 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-root\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733778 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733778 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-textfile\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733778 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-metrics-client-ca\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733778 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h94gd\" (UniqueName: \"kubernetes.io/projected/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-kube-api-access-h94gd\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733988 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-wtmp\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733988 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733988 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-sys\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.733988 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.733926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-tls\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-sys\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-tls\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-root\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-textfile\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-metrics-client-ca\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h94gd\" (UniqueName: \"kubernetes.io/projected/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-kube-api-access-h94gd\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-wtmp\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.835826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.835577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-wtmp\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.838102 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.836139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.838102 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.836203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-sys\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.838102 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:57.836286    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 20:50:57.838102 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.836581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-textfile\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.838102 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.836634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-root\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.838102 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.837758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-metrics-client-ca\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.838102 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:50:57.837883    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-tls podName:0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:58.33786513 +0000 UTC m=+98.031674446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-tls") pod "node-exporter-vl9j4" (UID: "0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e") : secret "node-exporter-tls" not found
Sep 17 20:50:57.846180 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.846063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:57.854841 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:57.854770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h94gd\" (UniqueName: \"kubernetes.io/projected/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-kube-api-access-h94gd\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:58.339814 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:58.339762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-tls\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:58.342597 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:58.342566    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e-node-exporter-tls\") pod \"node-exporter-vl9j4\" (UID: \"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e\") " pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:58.553159 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:58.553101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vl9j4"
Sep 17 20:50:59.600302 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.600270    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-869b875586-7s7pw"]
Sep 17 20:50:59.604731 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.604708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.607132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.606948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 20:50:59.607132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.606978    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 20:50:59.607132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.607011    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sxzqz\""
Sep 17 20:50:59.607465 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.607183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 20:50:59.607465 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.607311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 20:50:59.607716 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.607697    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e3b3drtjheoga\""
Sep 17 20:50:59.607816 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.607697    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 20:50:59.616541 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.616520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-869b875586-7s7pw"]
Sep 17 20:50:59.752218 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-tls\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.752417 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.752417 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.752417 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.752417 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-grpc-tls\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.752417 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c197e55-ac2a-495f-8ac4-328936334b88-metrics-client-ca\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.752666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.752666 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.752516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljc8\" (UniqueName: \"kubernetes.io/projected/4c197e55-ac2a-495f-8ac4-328936334b88-kube-api-access-jljc8\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853278 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-tls\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853278 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853278 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853575 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853575 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-grpc-tls\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853575 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c197e55-ac2a-495f-8ac4-328936334b88-metrics-client-ca\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853575 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.853575 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.853452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljc8\" (UniqueName: \"kubernetes.io/projected/4c197e55-ac2a-495f-8ac4-328936334b88-kube-api-access-jljc8\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.854722 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.854671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c197e55-ac2a-495f-8ac4-328936334b88-metrics-client-ca\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.856697 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.856660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.856826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.856756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-tls\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.857234 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.857205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.857377 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.857247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.857377 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.857297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.857737 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.857713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4c197e55-ac2a-495f-8ac4-328936334b88-secret-grpc-tls\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.862183 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.862164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljc8\" (UniqueName: \"kubernetes.io/projected/4c197e55-ac2a-495f-8ac4-328936334b88-kube-api-access-jljc8\") pod \"thanos-querier-869b875586-7s7pw\" (UID: \"4c197e55-ac2a-495f-8ac4-328936334b88\") " pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:50:59.916880 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:50:59.916847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:51:02.594601 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:51:02.594564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f19e1b5_bcb0_4f59_bf0e_d4a00c5c596e.slice/crio-15fc07fd32679e185f35d8eee6d6a0a3427fa24715dacb3062fb7b02f7164218 WatchSource:0}: Error finding container 15fc07fd32679e185f35d8eee6d6a0a3427fa24715dacb3062fb7b02f7164218: Status 404 returned error can't find the container with id 15fc07fd32679e185f35d8eee6d6a0a3427fa24715dacb3062fb7b02f7164218
Sep 17 20:51:02.738437 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.738406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl9j4" event={"ID":"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e","Type":"ContainerStarted","Data":"15fc07fd32679e185f35d8eee6d6a0a3427fa24715dacb3062fb7b02f7164218"}
Sep 17 20:51:02.840392 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.840299    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5684b66bd9-dbs92"]
Sep 17 20:51:02.859178 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.859140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5684b66bd9-dbs92"]
Sep 17 20:51:02.859356 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.859283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.861966 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.861937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 20:51:02.862200 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.862185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 20:51:02.862865 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.862425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 20:51:02.862865 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.862456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 20:51:02.862865 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.862603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 20:51:02.862865 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.862846    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-slg5l\""
Sep 17 20:51:02.863911 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.863880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-869b875586-7s7pw"]
Sep 17 20:51:02.869585 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.869566    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 20:51:02.935169 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:51:02.935136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c197e55_ac2a_495f_8ac4_328936334b88.slice/crio-6966b73bfc5df51fff9cf7301ecf6dc297469775685682ef25ca77e0d84735dd WatchSource:0}: Error finding container 6966b73bfc5df51fff9cf7301ecf6dc297469775685682ef25ca77e0d84735dd: Status 404 returned error can't find the container with id 6966b73bfc5df51fff9cf7301ecf6dc297469775685682ef25ca77e0d84735dd
Sep 17 20:51:02.981636 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-federate-client-tls\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.981769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.981769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-metrics-client-ca\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.981769 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-telemeter-client-tls\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.981919 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-serving-certs-ca-bundle\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.981919 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzng\" (UniqueName: \"kubernetes.io/projected/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-kube-api-access-czzng\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.981919 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-secret-telemeter-client\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:02.982022 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:02.981938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082422 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082566 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-federate-client-tls\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082566 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082566 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-metrics-client-ca\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082566 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-telemeter-client-tls\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082783 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-serving-certs-ca-bundle\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082783 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzng\" (UniqueName: \"kubernetes.io/projected/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-kube-api-access-czzng\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.082887 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.082813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-secret-telemeter-client\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.083343 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.083268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-metrics-client-ca\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.083343 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.083281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.083571 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.083525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-serving-certs-ca-bundle\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.085852 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.085811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-federate-client-tls\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.085852 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.085821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.086001 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.085983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-telemeter-client-tls\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.086358 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.086314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-secret-telemeter-client\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.093140 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.093117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzng\" (UniqueName: \"kubernetes.io/projected/0d5bdeea-8fe2-4fde-9fa3-85492620a4cf-kube-api-access-czzng\") pod \"telemeter-client-5684b66bd9-dbs92\" (UID: \"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf\") " pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.171476 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.171440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92"
Sep 17 20:51:03.471558 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.471431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5684b66bd9-dbs92"]
Sep 17 20:51:03.615303 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:51:03.615263    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5bdeea_8fe2_4fde_9fa3_85492620a4cf.slice/crio-898a4f4e2da62b68d250af927830d6760e11ea3075d9f16da882e96081c48289 WatchSource:0}: Error finding container 898a4f4e2da62b68d250af927830d6760e11ea3075d9f16da882e96081c48289: Status 404 returned error can't find the container with id 898a4f4e2da62b68d250af927830d6760e11ea3075d9f16da882e96081c48289
Sep 17 20:51:03.744953 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.744859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-h5nzh" event={"ID":"32dd64bc-fb24-498d-a947-deb5e66032b9","Type":"ContainerStarted","Data":"a59cde8d32d7ea3771ad109f73afc46748080d9fac4945a19c58298e3d15fdb2"}
Sep 17 20:51:03.747680 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.745912    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-h5nzh"
Sep 17 20:51:03.747680 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.747630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92" event={"ID":"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf","Type":"ContainerStarted","Data":"898a4f4e2da62b68d250af927830d6760e11ea3075d9f16da882e96081c48289"}
Sep 17 20:51:03.749212 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.749146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" event={"ID":"4c197e55-ac2a-495f-8ac4-328936334b88","Type":"ContainerStarted","Data":"6966b73bfc5df51fff9cf7301ecf6dc297469775685682ef25ca77e0d84735dd"}
Sep 17 20:51:03.757158 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.757129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-h5nzh"
Sep 17 20:51:03.770743 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.769930    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-h5nzh" podStartSLOduration=2.271185807 podStartE2EDuration="20.769910158s" totalImagesPullingTime="18.498724351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:03.76869605 +0000 UTC m=+103.462505382" watchObservedRunningTime="2026-09-17 20:51:03.769910158 +0000 UTC m=+103.463719490"
Sep 17 20:51:03.912262 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:03.912225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:51:04.611069 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:04.611041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r5xpp"
Sep 17 20:51:04.757444 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:04.756737    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e" containerID="735b2f43a33a9e5b5c319f724b27ae846f6502f8c498d1c6a0d4cef6b71c7977" exitCode=0
Sep 17 20:51:04.759139 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:04.759080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl9j4" event={"ID":"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e","Type":"ContainerDied","Data":"735b2f43a33a9e5b5c319f724b27ae846f6502f8c498d1c6a0d4cef6b71c7977"}
Sep 17 20:51:05.764461 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:05.764195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl9j4" event={"ID":"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e","Type":"ContainerStarted","Data":"cc68e3c26c9a9e8c5aeef679f70758426d036d6db6ac3568aee16813f80c90f2"}
Sep 17 20:51:05.764461 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:05.764257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl9j4" event={"ID":"0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e","Type":"ContainerStarted","Data":"108daa5e9d9964a68a6d937be0051eeb081067f3f3c76cb70673fc0b7aa1019a"}
Sep 17 20:51:05.786353 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:05.786285    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vl9j4" podStartSLOduration=7.2698791830000005 podStartE2EDuration="8.78626366s" totalImagesPullingTime="1.414992864s" totalInitContainerRuntime="101.391613ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:05.785293064 +0000 UTC m=+105.479102419" watchObservedRunningTime="2026-09-17 20:51:05.78626366 +0000 UTC m=+105.480072993"
Sep 17 20:51:06.671409 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:06.671382    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6445b7fc9f-gj4gh"
Sep 17 20:51:07.063355 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.063263    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-846fb6f5d4-27hvd"]
Sep 17 20:51:07.083388 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.083352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846fb6f5d4-27hvd"]
Sep 17 20:51:07.083548 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.083442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.086469 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.086201    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5tlfx\""
Sep 17 20:51:07.086595 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.086491    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 20:51:07.086595 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.086515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 20:51:07.087111 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.086889    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 20:51:07.087111 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.086900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 20:51:07.089204 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.089180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 20:51:07.094636 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.094618    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 20:51:07.236442 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.236414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-service-ca\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.236561 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.236454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-oauth-serving-cert\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.236561 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.236492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-trusted-ca-bundle\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.236561 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.236521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-oauth-config\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.236711 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.236599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-config\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.236711 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.236624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgzw\" (UniqueName: \"kubernetes.io/projected/09e0a84e-107e-4f62-8664-75bce57b3a7f-kube-api-access-xbgzw\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.236711 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.236679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-serving-cert\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338037 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.337055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-config\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338037 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.337093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgzw\" (UniqueName: \"kubernetes.io/projected/09e0a84e-107e-4f62-8664-75bce57b3a7f-kube-api-access-xbgzw\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338037 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.337142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-serving-cert\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338037 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.337185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-service-ca\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338037 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.337208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-oauth-serving-cert\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338037 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.337238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-trusted-ca-bundle\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338037 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.337264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-oauth-config\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338549 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.338478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-config\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.338656 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.338636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-oauth-serving-cert\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.339348 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.339073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-service-ca\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.339904 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.339734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-trusted-ca-bundle\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.340399 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.340194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-oauth-config\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.340746 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.340724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-serving-cert\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.347858 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.347836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgzw\" (UniqueName: \"kubernetes.io/projected/09e0a84e-107e-4f62-8664-75bce57b3a7f-kube-api-access-xbgzw\") pod \"console-846fb6f5d4-27hvd\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") " pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.396598 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.396570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:07.701247 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.701211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846fb6f5d4-27hvd"]
Sep 17 20:51:07.706291 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:51:07.706264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e0a84e_107e_4f62_8664_75bce57b3a7f.slice/crio-d4e445e5c2d81399dfa19a337de3f9fe74c0158d66f383f85c082fd2c6a0a5e5 WatchSource:0}: Error finding container d4e445e5c2d81399dfa19a337de3f9fe74c0158d66f383f85c082fd2c6a0a5e5: Status 404 returned error can't find the container with id d4e445e5c2d81399dfa19a337de3f9fe74c0158d66f383f85c082fd2c6a0a5e5
Sep 17 20:51:07.776619 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.776582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92" event={"ID":"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf","Type":"ContainerStarted","Data":"55676b96db1102e29ddaa462d0bc67173dfadf1e5355832c034cd7c5ce5e7ace"}
Sep 17 20:51:07.780619 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.780584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" event={"ID":"4c197e55-ac2a-495f-8ac4-328936334b88","Type":"ContainerStarted","Data":"9fa41f7b57f9c81064f3cc72bf423a758c2ac8be1e0279f3b66e86cace035c6c"}
Sep 17 20:51:07.780755 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.780664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" event={"ID":"4c197e55-ac2a-495f-8ac4-328936334b88","Type":"ContainerStarted","Data":"f6bafac2e822cbc578e7407352a7ea987dd98d7d9004979f006ee53652ac3e57"}
Sep 17 20:51:07.780755 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.780700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" event={"ID":"4c197e55-ac2a-495f-8ac4-328936334b88","Type":"ContainerStarted","Data":"96ca3b3cda3d8d2a3f26a6a46fecbf3662e03e9fedba698b8c3780dcd376a7e4"}
Sep 17 20:51:07.782460 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:07.782428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846fb6f5d4-27hvd" event={"ID":"09e0a84e-107e-4f62-8664-75bce57b3a7f","Type":"ContainerStarted","Data":"d4e445e5c2d81399dfa19a337de3f9fe74c0158d66f383f85c082fd2c6a0a5e5"}
Sep 17 20:51:08.926615 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:08.926425    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64768c58f6-clv7j" podUID="ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" containerName="registry" containerID="cri-o://42251411127fc3c1e9882a42abd25201a92979dedebcd50d2eb5c2ae5a2f5b30" gracePeriod=30
Sep 17 20:51:09.800339 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:09.800276    2579 generic.go:356] "Generic (PLEG): container finished" podID="ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" containerID="42251411127fc3c1e9882a42abd25201a92979dedebcd50d2eb5c2ae5a2f5b30" exitCode=0
Sep 17 20:51:09.800503 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:09.800411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64768c58f6-clv7j" event={"ID":"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d","Type":"ContainerDied","Data":"42251411127fc3c1e9882a42abd25201a92979dedebcd50d2eb5c2ae5a2f5b30"}
Sep 17 20:51:09.916096 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:09.915987    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-certificates\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-ca-trust-extracted\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071691    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-trusted-ca\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-bound-sa-token\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4cnn\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-kube-api-access-p4cnn\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-installation-pull-secrets\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.072849 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.071889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-image-registry-private-configuration\") pod \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\" (UID: \"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d\") "
Sep 17 20:51:10.074194 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.074156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:10.075811 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.075763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:10.087034 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.086907    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:51:10.087196 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.086940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:51:10.087280 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.087200    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:51:10.087674 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.087624    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:51:10.087900 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.087866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-kube-api-access-p4cnn" (OuterVolumeSpecName: "kube-api-access-p4cnn") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "kube-api-access-p4cnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:51:10.089110 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.088742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" (UID: "ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:51:10.173160 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173120    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-installation-pull-secrets\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.173160 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173163    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-image-registry-private-configuration\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.173425 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173181    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-certificates\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.173425 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173196    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-ca-trust-extracted\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.173425 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173209    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-trusted-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.173425 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173222    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-registry-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.173425 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173235    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-bound-sa-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.173425 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.173247    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4cnn\" (UniqueName: \"kubernetes.io/projected/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d-kube-api-access-p4cnn\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:10.809843 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.808867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64768c58f6-clv7j" event={"ID":"ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d","Type":"ContainerDied","Data":"665ba352f6713667691ab10b09d1abee31c5b3167008582aa64c8ccf4fe6bef5"}
Sep 17 20:51:10.809843 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.808937    2579 scope.go:117] "RemoveContainer" containerID="42251411127fc3c1e9882a42abd25201a92979dedebcd50d2eb5c2ae5a2f5b30"
Sep 17 20:51:10.809843 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.809129    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64768c58f6-clv7j"
Sep 17 20:51:10.829886 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.829857    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64768c58f6-clv7j"]
Sep 17 20:51:10.833707 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:10.833682    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64768c58f6-clv7j"]
Sep 17 20:51:12.298367 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:12.298339    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" path="/var/lib/kubelet/pods/ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d/volumes"
Sep 17 20:51:13.824800 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:13.824769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92" event={"ID":"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf","Type":"ContainerStarted","Data":"b3e8ce4db8399638887695d229febda1c69ab2d94c176c743f6b039f03438e6b"}
Sep 17 20:51:13.825216 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:13.824821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92" event={"ID":"0d5bdeea-8fe2-4fde-9fa3-85492620a4cf","Type":"ContainerStarted","Data":"e9dc318774404a31dc2c97110c57ca26518ef74cd388f9a3adfb2de6a34f68f5"}
Sep 17 20:51:13.827508 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:13.827485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" event={"ID":"4c197e55-ac2a-495f-8ac4-328936334b88","Type":"ContainerStarted","Data":"ab28831f16a85c1c5004fab2a0f650254c7ab27d3a78dd14566117df2f0dfe32"}
Sep 17 20:51:13.827629 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:13.827542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" event={"ID":"4c197e55-ac2a-495f-8ac4-328936334b88","Type":"ContainerStarted","Data":"ba1231b2431009ff1986bc67d5e84377953bcd9de8e2facb04b8efaf30393d74"}
Sep 17 20:51:13.857209 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:13.857151    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5684b66bd9-dbs92" podStartSLOduration=2.097007027 podStartE2EDuration="11.857138152s" totalImagesPullingTime="9.760131125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:13.855798439 +0000 UTC m=+113.549607789" watchObservedRunningTime="2026-09-17 20:51:13.857138152 +0000 UTC m=+113.550947483"
Sep 17 20:51:13.966113 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:13.966016    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846fb6f5d4-27hvd"]
Sep 17 20:51:14.001199 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.001177    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fc484954b-j8tc6"]
Sep 17 20:51:14.001523 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.001510    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" containerName="registry"
Sep 17 20:51:14.001577 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.001525    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" containerName="registry"
Sep 17 20:51:14.001577 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.001574    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb0440a-2cb0-4e1d-833c-bb3feccc2e9d" containerName="registry"
Sep 17 20:51:14.022347 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.022248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc484954b-j8tc6"]
Sep 17 20:51:14.022502 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.022381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.113129 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.113100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctcgk\" (UniqueName: \"kubernetes.io/projected/0bb36050-d1d5-4d1c-a678-208d50035b81-kube-api-access-ctcgk\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.113295 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.113151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-trusted-ca-bundle\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.113295 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.113201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-console-config\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.113295 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.113258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-service-ca\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.113295 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.113277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-oauth-serving-cert\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.113459 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.113299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-oauth-config\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.113459 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.113351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-serving-cert\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.213865 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.213828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-trusted-ca-bundle\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.213865 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.213863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-console-config\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.214108 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.213890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-service-ca\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.214108 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.213909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-oauth-serving-cert\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.214108 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.213936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-oauth-config\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.214108 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.213967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-serving-cert\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.214108 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.214028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctcgk\" (UniqueName: \"kubernetes.io/projected/0bb36050-d1d5-4d1c-a678-208d50035b81-kube-api-access-ctcgk\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.214719 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.214691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-service-ca\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.215070 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.214793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-trusted-ca-bundle\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.215070 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.214793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-console-config\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.215202 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.214814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-oauth-serving-cert\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.216509 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.216489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-oauth-config\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.216621 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.216602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-serving-cert\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.222621 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.222603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctcgk\" (UniqueName: \"kubernetes.io/projected/0bb36050-d1d5-4d1c-a678-208d50035b81-kube-api-access-ctcgk\") pod \"console-fc484954b-j8tc6\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") " pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.334661 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.334628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:14.626412 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.626391    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc484954b-j8tc6"]
Sep 17 20:51:14.627943 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:51:14.627915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb36050_d1d5_4d1c_a678_208d50035b81.slice/crio-d9e0a3cec0ead38ab90128db0f70700f6adf23489661c829abbe77064964af8e WatchSource:0}: Error finding container d9e0a3cec0ead38ab90128db0f70700f6adf23489661c829abbe77064964af8e: Status 404 returned error can't find the container with id d9e0a3cec0ead38ab90128db0f70700f6adf23489661c829abbe77064964af8e
Sep 17 20:51:14.835609 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.835566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" event={"ID":"4c197e55-ac2a-495f-8ac4-328936334b88","Type":"ContainerStarted","Data":"340a3abc960f414810af47af36dc4046c41ecccea6d9fc83ee3cf546f8ae8302"}
Sep 17 20:51:14.836121 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.835817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:51:14.837460 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.837430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846fb6f5d4-27hvd" event={"ID":"09e0a84e-107e-4f62-8664-75bce57b3a7f","Type":"ContainerStarted","Data":"8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4"}
Sep 17 20:51:14.839487 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.839432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc484954b-j8tc6" event={"ID":"0bb36050-d1d5-4d1c-a678-208d50035b81","Type":"ContainerStarted","Data":"59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e"}
Sep 17 20:51:14.839606 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.839512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc484954b-j8tc6" event={"ID":"0bb36050-d1d5-4d1c-a678-208d50035b81","Type":"ContainerStarted","Data":"d9e0a3cec0ead38ab90128db0f70700f6adf23489661c829abbe77064964af8e"}
Sep 17 20:51:14.841441 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.841416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw"
Sep 17 20:51:14.875122 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.875075    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-869b875586-7s7pw" podStartSLOduration=5.775366527 podStartE2EDuration="15.875060935s" totalImagesPullingTime="10.099694408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:14.874448862 +0000 UTC m=+114.568258208" watchObservedRunningTime="2026-09-17 20:51:14.875060935 +0000 UTC m=+114.568870321"
Sep 17 20:51:14.898228 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.898100    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-846fb6f5d4-27hvd" podStartSLOduration=1.710337035 podStartE2EDuration="7.898080526s" totalImagesPullingTime="6.187743491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:14.897908893 +0000 UTC m=+114.591718242" watchObservedRunningTime="2026-09-17 20:51:14.898080526 +0000 UTC m=+114.591889858"
Sep 17 20:51:14.917488 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:14.917441    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fc484954b-j8tc6" podStartSLOduration=1.917428107 podStartE2EDuration="1.917428107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:14.915878653 +0000 UTC m=+114.609687985" watchObservedRunningTime="2026-09-17 20:51:14.917428107 +0000 UTC m=+114.611237438"
Sep 17 20:51:17.397795 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:17.397758    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:24.335364 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:24.335315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:24.335817 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:24.335388    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:24.340197 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:24.340175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:24.881891 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:24.881865    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:51:32.055132 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:32.055095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:51:32.057543 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:32.057523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410d7742-1dd0-4ca9-abcf-3b999c572cbf-metrics-certs\") pod \"network-metrics-daemon-sn8wn\" (UID: \"410d7742-1dd0-4ca9-abcf-3b999c572cbf\") " pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:51:32.313160 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:32.313093    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-775ms\""
Sep 17 20:51:32.321252 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:32.321236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn8wn"
Sep 17 20:51:32.611055 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:32.611026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sn8wn"]
Sep 17 20:51:32.614421 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:51:32.614399    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410d7742_1dd0_4ca9_abcf_3b999c572cbf.slice/crio-16b41e621af00e24d935f70ecbd34cbf83843ff0e4d9d677bcc8bed2dda9720c WatchSource:0}: Error finding container 16b41e621af00e24d935f70ecbd34cbf83843ff0e4d9d677bcc8bed2dda9720c: Status 404 returned error can't find the container with id 16b41e621af00e24d935f70ecbd34cbf83843ff0e4d9d677bcc8bed2dda9720c
Sep 17 20:51:32.908042 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:32.907959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sn8wn" event={"ID":"410d7742-1dd0-4ca9-abcf-3b999c572cbf","Type":"ContainerStarted","Data":"16b41e621af00e24d935f70ecbd34cbf83843ff0e4d9d677bcc8bed2dda9720c"}
Sep 17 20:51:34.916864 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:34.916826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sn8wn" event={"ID":"410d7742-1dd0-4ca9-abcf-3b999c572cbf","Type":"ContainerStarted","Data":"734202bf347f15f1b67f03ca2a20f829012305e7fc9f6ab2377eeedfaba1d5cd"}
Sep 17 20:51:34.917253 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:34.916884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sn8wn" event={"ID":"410d7742-1dd0-4ca9-abcf-3b999c572cbf","Type":"ContainerStarted","Data":"93837b6b0505db095dd81d15bcaf9e584a505f7e662a53d7cdef99ff15c2d235"}
Sep 17 20:51:34.937188 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:34.937135    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sn8wn" podStartSLOduration=131.631440597 podStartE2EDuration="2m12.9371216s" totalImagesPullingTime="1.305681003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:34.935758498 +0000 UTC m=+134.629567847" watchObservedRunningTime="2026-09-17 20:51:34.9371216 +0000 UTC m=+134.630930930"
Sep 17 20:51:39.860441 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:39.860402    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-846fb6f5d4-27hvd" podUID="09e0a84e-107e-4f62-8664-75bce57b3a7f" containerName="console" containerID="cri-o://8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4" gracePeriod=15
Sep 17 20:51:40.138144 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.138086    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:40.224483 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-trusted-ca-bundle\") pod \"09e0a84e-107e-4f62-8664-75bce57b3a7f\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") "
Sep 17 20:51:40.224658 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-serving-cert\") pod \"09e0a84e-107e-4f62-8664-75bce57b3a7f\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") "
Sep 17 20:51:40.224658 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-service-ca\") pod \"09e0a84e-107e-4f62-8664-75bce57b3a7f\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") "
Sep 17 20:51:40.224658 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbgzw\" (UniqueName: \"kubernetes.io/projected/09e0a84e-107e-4f62-8664-75bce57b3a7f-kube-api-access-xbgzw\") pod \"09e0a84e-107e-4f62-8664-75bce57b3a7f\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") "
Sep 17 20:51:40.224658 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-oauth-serving-cert\") pod \"09e0a84e-107e-4f62-8664-75bce57b3a7f\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") "
Sep 17 20:51:40.224868 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-oauth-config\") pod \"09e0a84e-107e-4f62-8664-75bce57b3a7f\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") "
Sep 17 20:51:40.224868 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-config\") pod \"09e0a84e-107e-4f62-8664-75bce57b3a7f\" (UID: \"09e0a84e-107e-4f62-8664-75bce57b3a7f\") "
Sep 17 20:51:40.224977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-service-ca" (OuterVolumeSpecName: "service-ca") pod "09e0a84e-107e-4f62-8664-75bce57b3a7f" (UID: "09e0a84e-107e-4f62-8664-75bce57b3a7f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:40.224977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.224943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09e0a84e-107e-4f62-8664-75bce57b3a7f" (UID: "09e0a84e-107e-4f62-8664-75bce57b3a7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:40.225065 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.225032    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09e0a84e-107e-4f62-8664-75bce57b3a7f" (UID: "09e0a84e-107e-4f62-8664-75bce57b3a7f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:40.225239 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.225217    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-config" (OuterVolumeSpecName: "console-config") pod "09e0a84e-107e-4f62-8664-75bce57b3a7f" (UID: "09e0a84e-107e-4f62-8664-75bce57b3a7f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:40.227059 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.227030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09e0a84e-107e-4f62-8664-75bce57b3a7f" (UID: "09e0a84e-107e-4f62-8664-75bce57b3a7f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:51:40.227059 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.227038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09e0a84e-107e-4f62-8664-75bce57b3a7f" (UID: "09e0a84e-107e-4f62-8664-75bce57b3a7f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:51:40.227182 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.227123    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e0a84e-107e-4f62-8664-75bce57b3a7f-kube-api-access-xbgzw" (OuterVolumeSpecName: "kube-api-access-xbgzw") pod "09e0a84e-107e-4f62-8664-75bce57b3a7f" (UID: "09e0a84e-107e-4f62-8664-75bce57b3a7f"). InnerVolumeSpecName "kube-api-access-xbgzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:51:40.325947 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.325921    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:40.325947 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.325945    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:40.326090 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.325956    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbgzw\" (UniqueName: \"kubernetes.io/projected/09e0a84e-107e-4f62-8664-75bce57b3a7f-kube-api-access-xbgzw\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:40.326090 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.325970    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-oauth-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:40.326090 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.325979    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-oauth-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:40.326090 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.325987    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-console-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:40.326090 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.325995    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09e0a84e-107e-4f62-8664-75bce57b3a7f-trusted-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:51:40.941848 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.941810    2579 generic.go:356] "Generic (PLEG): container finished" podID="09e0a84e-107e-4f62-8664-75bce57b3a7f" containerID="8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4" exitCode=0
Sep 17 20:51:40.942223 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.941858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846fb6f5d4-27hvd"
Sep 17 20:51:40.942223 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.941897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846fb6f5d4-27hvd" event={"ID":"09e0a84e-107e-4f62-8664-75bce57b3a7f","Type":"ContainerDied","Data":"8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4"}
Sep 17 20:51:40.942223 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.941950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846fb6f5d4-27hvd" event={"ID":"09e0a84e-107e-4f62-8664-75bce57b3a7f","Type":"ContainerDied","Data":"d4e445e5c2d81399dfa19a337de3f9fe74c0158d66f383f85c082fd2c6a0a5e5"}
Sep 17 20:51:40.942223 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.941970    2579 scope.go:117] "RemoveContainer" containerID="8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4"
Sep 17 20:51:40.950396 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.950376    2579 scope.go:117] "RemoveContainer" containerID="8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4"
Sep 17 20:51:40.950704 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:51:40.950684    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4\": container with ID starting with 8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4 not found: ID does not exist" containerID="8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4"
Sep 17 20:51:40.950768 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.950711    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4"} err="failed to get container status \"8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4\": rpc error: code = NotFound desc = could not find container \"8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4\": container with ID starting with 8b497c1da77d1be7e6f11cffe93a7b188825e9c9ae84a4ecab277c932defdea4 not found: ID does not exist"
Sep 17 20:51:40.957916 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.957890    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846fb6f5d4-27hvd"]
Sep 17 20:51:40.960599 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:40.960579    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-846fb6f5d4-27hvd"]
Sep 17 20:51:42.300017 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:51:42.299982    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e0a84e-107e-4f62-8664-75bce57b3a7f" path="/var/lib/kubelet/pods/09e0a84e-107e-4f62-8664-75bce57b3a7f/volumes"
Sep 17 20:52:33.579058 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.579020    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57948746bf-2tmcw"]
Sep 17 20:52:33.579514 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.579365    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e0a84e-107e-4f62-8664-75bce57b3a7f" containerName="console"
Sep 17 20:52:33.579514 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.579382    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e0a84e-107e-4f62-8664-75bce57b3a7f" containerName="console"
Sep 17 20:52:33.579514 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.579431    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e0a84e-107e-4f62-8664-75bce57b3a7f" containerName="console"
Sep 17 20:52:33.582420 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.582393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.598577 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.598548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57948746bf-2tmcw"]
Sep 17 20:52:33.639727 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.639686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-serving-cert\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.639904 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.639741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-oauth-config\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.639904 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.639763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-config\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.639904 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.639781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxc5\" (UniqueName: \"kubernetes.io/projected/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-kube-api-access-zsxc5\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.639904 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.639804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-service-ca\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.639904 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.639837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-oauth-serving-cert\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.640110 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.639915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-trusted-ca-bundle\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.740683 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.740644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-serving-cert\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.740683 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.740682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-oauth-config\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.740925 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.740701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-config\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.740925 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.740779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxc5\" (UniqueName: \"kubernetes.io/projected/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-kube-api-access-zsxc5\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.740925 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.740831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-service-ca\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.740925 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.740866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-oauth-serving-cert\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.740925 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.740904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-trusted-ca-bundle\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.741457 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.741432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-config\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.741583 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.741556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-oauth-serving-cert\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.741655 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.741631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-service-ca\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.742108 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.742086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-trusted-ca-bundle\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.743455 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.743431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-serving-cert\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.743542 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.743490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-oauth-config\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.749598 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.749575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxc5\" (UniqueName: \"kubernetes.io/projected/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-kube-api-access-zsxc5\") pod \"console-57948746bf-2tmcw\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") " pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:33.892873 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:33.892793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:34.166000 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:34.165926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57948746bf-2tmcw"]
Sep 17 20:52:34.169135 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:52:34.169103    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbccfb2e_98b4_4c5f_8bc2_ed7c4d293ed0.slice/crio-5f8daf12e0a10e63b2947915f55cb009ff39eabe67e4232868fc9f31656536b4 WatchSource:0}: Error finding container 5f8daf12e0a10e63b2947915f55cb009ff39eabe67e4232868fc9f31656536b4: Status 404 returned error can't find the container with id 5f8daf12e0a10e63b2947915f55cb009ff39eabe67e4232868fc9f31656536b4
Sep 17 20:52:35.162523 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:35.162488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57948746bf-2tmcw" event={"ID":"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0","Type":"ContainerStarted","Data":"595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5"}
Sep 17 20:52:35.162900 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:35.162536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57948746bf-2tmcw" event={"ID":"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0","Type":"ContainerStarted","Data":"5f8daf12e0a10e63b2947915f55cb009ff39eabe67e4232868fc9f31656536b4"}
Sep 17 20:52:35.187293 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:35.187242    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57948746bf-2tmcw" podStartSLOduration=2.187227941 podStartE2EDuration="2.187227941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:35.186296636 +0000 UTC m=+194.880105978" watchObservedRunningTime="2026-09-17 20:52:35.187227941 +0000 UTC m=+194.881037271"
Sep 17 20:52:43.893205 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:43.893167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:43.893690 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:43.893221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:43.897948 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:43.897927    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:44.197943 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:44.197862    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:52:44.247040 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:52:44.247007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc484954b-j8tc6"]
Sep 17 20:53:09.267176 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.267129    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fc484954b-j8tc6" podUID="0bb36050-d1d5-4d1c-a678-208d50035b81" containerName="console" containerID="cri-o://59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e" gracePeriod=15
Sep 17 20:53:09.512033 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.512009    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:53:09.627332 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627300    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctcgk\" (UniqueName: \"kubernetes.io/projected/0bb36050-d1d5-4d1c-a678-208d50035b81-kube-api-access-ctcgk\") pod \"0bb36050-d1d5-4d1c-a678-208d50035b81\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") "
Sep 17 20:53:09.627495 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627392    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-serving-cert\") pod \"0bb36050-d1d5-4d1c-a678-208d50035b81\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") "
Sep 17 20:53:09.627495 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-console-config\") pod \"0bb36050-d1d5-4d1c-a678-208d50035b81\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") "
Sep 17 20:53:09.627495 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-service-ca\") pod \"0bb36050-d1d5-4d1c-a678-208d50035b81\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") "
Sep 17 20:53:09.627640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-oauth-serving-cert\") pod \"0bb36050-d1d5-4d1c-a678-208d50035b81\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") "
Sep 17 20:53:09.627640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-trusted-ca-bundle\") pod \"0bb36050-d1d5-4d1c-a678-208d50035b81\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") "
Sep 17 20:53:09.627640 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-oauth-config\") pod \"0bb36050-d1d5-4d1c-a678-208d50035b81\" (UID: \"0bb36050-d1d5-4d1c-a678-208d50035b81\") "
Sep 17 20:53:09.627951 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627915    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-console-config" (OuterVolumeSpecName: "console-config") pod "0bb36050-d1d5-4d1c-a678-208d50035b81" (UID: "0bb36050-d1d5-4d1c-a678-208d50035b81"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:09.628068 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0bb36050-d1d5-4d1c-a678-208d50035b81" (UID: "0bb36050-d1d5-4d1c-a678-208d50035b81"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:09.628068 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627951    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-service-ca" (OuterVolumeSpecName: "service-ca") pod "0bb36050-d1d5-4d1c-a678-208d50035b81" (UID: "0bb36050-d1d5-4d1c-a678-208d50035b81"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:09.628068 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.627971    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0bb36050-d1d5-4d1c-a678-208d50035b81" (UID: "0bb36050-d1d5-4d1c-a678-208d50035b81"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:09.629757 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.629725    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0bb36050-d1d5-4d1c-a678-208d50035b81" (UID: "0bb36050-d1d5-4d1c-a678-208d50035b81"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:09.629757 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.629734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb36050-d1d5-4d1c-a678-208d50035b81-kube-api-access-ctcgk" (OuterVolumeSpecName: "kube-api-access-ctcgk") pod "0bb36050-d1d5-4d1c-a678-208d50035b81" (UID: "0bb36050-d1d5-4d1c-a678-208d50035b81"). InnerVolumeSpecName "kube-api-access-ctcgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:09.629880 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.629758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0bb36050-d1d5-4d1c-a678-208d50035b81" (UID: "0bb36050-d1d5-4d1c-a678-208d50035b81"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:09.728960 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.728927    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:53:09.728960 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.728952    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-console-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:53:09.728960 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.728963    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:53:09.729176 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.728971    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-oauth-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:53:09.729176 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.728980    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb36050-d1d5-4d1c-a678-208d50035b81-trusted-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:53:09.729176 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.728987    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bb36050-d1d5-4d1c-a678-208d50035b81-console-oauth-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:53:09.729176 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:09.728997    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctcgk\" (UniqueName: \"kubernetes.io/projected/0bb36050-d1d5-4d1c-a678-208d50035b81-kube-api-access-ctcgk\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:53:10.290394 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.290364    2579 generic.go:356] "Generic (PLEG): container finished" podID="0bb36050-d1d5-4d1c-a678-208d50035b81" containerID="59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e" exitCode=0
Sep 17 20:53:10.290394 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.290398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc484954b-j8tc6" event={"ID":"0bb36050-d1d5-4d1c-a678-208d50035b81","Type":"ContainerDied","Data":"59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e"}
Sep 17 20:53:10.290867 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.290407    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc484954b-j8tc6"
Sep 17 20:53:10.290867 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.290431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc484954b-j8tc6" event={"ID":"0bb36050-d1d5-4d1c-a678-208d50035b81","Type":"ContainerDied","Data":"d9e0a3cec0ead38ab90128db0f70700f6adf23489661c829abbe77064964af8e"}
Sep 17 20:53:10.290867 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.290450    2579 scope.go:117] "RemoveContainer" containerID="59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e"
Sep 17 20:53:10.299646 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.299630    2579 scope.go:117] "RemoveContainer" containerID="59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e"
Sep 17 20:53:10.299937 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:53:10.299912    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e\": container with ID starting with 59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e not found: ID does not exist" containerID="59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e"
Sep 17 20:53:10.300034 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.299942    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e"} err="failed to get container status \"59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e\": rpc error: code = NotFound desc = could not find container \"59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e\": container with ID starting with 59288a1a73d840eeedc530d9f66175d07751e2d9933dedcbfcc190ad6fcbd28e not found: ID does not exist"
Sep 17 20:53:10.311625 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.311603    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc484954b-j8tc6"]
Sep 17 20:53:10.315692 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:10.315673    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fc484954b-j8tc6"]
Sep 17 20:53:12.298979 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:12.298946    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb36050-d1d5-4d1c-a678-208d50035b81" path="/var/lib/kubelet/pods/0bb36050-d1d5-4d1c-a678-208d50035b81/volumes"
Sep 17 20:53:57.086588 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.086501    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"]
Sep 17 20:53:57.087067 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.086834    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb36050-d1d5-4d1c-a678-208d50035b81" containerName="console"
Sep 17 20:53:57.087067 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.086847    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb36050-d1d5-4d1c-a678-208d50035b81" containerName="console"
Sep 17 20:53:57.087067 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.086911    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb36050-d1d5-4d1c-a678-208d50035b81" containerName="console"
Sep 17 20:53:57.089630 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.089614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.092398 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.092374    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 20:53:57.092518 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.092375    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xdhdw\""
Sep 17 20:53:57.093006 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.092992    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 20:53:57.098415 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.098394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"]
Sep 17 20:53:57.169608 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.169576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.169776 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.169615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vlp\" (UniqueName: \"kubernetes.io/projected/3afd5324-220d-42aa-8aa8-d424badc6efe-kube-api-access-m8vlp\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.169776 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.169689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.270795 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.270763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vlp\" (UniqueName: \"kubernetes.io/projected/3afd5324-220d-42aa-8aa8-d424badc6efe-kube-api-access-m8vlp\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.270933 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.270814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.270933 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.270847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.271152 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.271137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.271188 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.271166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.279074 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.279046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vlp\" (UniqueName: \"kubernetes.io/projected/3afd5324-220d-42aa-8aa8-d424badc6efe-kube-api-access-m8vlp\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.399656 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.399575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:53:57.669417 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:57.669347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"]
Sep 17 20:53:57.672922 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:53:57.672897    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afd5324_220d_42aa_8aa8_d424badc6efe.slice/crio-3f6e857bfc92340dce812ad2e83ad75074e1c8cfb5ac00d73f78119276bc2986 WatchSource:0}: Error finding container 3f6e857bfc92340dce812ad2e83ad75074e1c8cfb5ac00d73f78119276bc2986: Status 404 returned error can't find the container with id 3f6e857bfc92340dce812ad2e83ad75074e1c8cfb5ac00d73f78119276bc2986
Sep 17 20:53:58.460940 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:53:58.460905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z" event={"ID":"3afd5324-220d-42aa-8aa8-d424badc6efe","Type":"ContainerStarted","Data":"3f6e857bfc92340dce812ad2e83ad75074e1c8cfb5ac00d73f78119276bc2986"}
Sep 17 20:54:04.487646 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:04.487609    2579 generic.go:356] "Generic (PLEG): container finished" podID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerID="81442eb49b2cf3f11e8fd5bbf36a4e7c8c993fe615d08eb620b273798bf8073a" exitCode=0
Sep 17 20:54:04.488027 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:04.487694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z" event={"ID":"3afd5324-220d-42aa-8aa8-d424badc6efe","Type":"ContainerDied","Data":"81442eb49b2cf3f11e8fd5bbf36a4e7c8c993fe615d08eb620b273798bf8073a"}
Sep 17 20:54:07.503099 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:07.503067    2579 generic.go:356] "Generic (PLEG): container finished" podID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerID="5e86fca90e19c59f1f7a464fcdaf43050fb5e0be06c5522df8e2a5b3c622e4d5" exitCode=0
Sep 17 20:54:07.503473 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:07.503162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z" event={"ID":"3afd5324-220d-42aa-8aa8-d424badc6efe","Type":"ContainerDied","Data":"5e86fca90e19c59f1f7a464fcdaf43050fb5e0be06c5522df8e2a5b3c622e4d5"}
Sep 17 20:54:14.538924 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:14.538895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z" event={"ID":"3afd5324-220d-42aa-8aa8-d424badc6efe","Type":"ContainerStarted","Data":"59630347b5c91ed8207ba5856a8f7b108107b521a0c7e24cabf4bff182776afe"}
Sep 17 20:54:15.544997 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.544961    2579 generic.go:356] "Generic (PLEG): container finished" podID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerID="59630347b5c91ed8207ba5856a8f7b108107b521a0c7e24cabf4bff182776afe" exitCode=0
Sep 17 20:54:15.545383 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.545051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z" event={"ID":"3afd5324-220d-42aa-8aa8-d424badc6efe","Type":"ContainerDied","Data":"59630347b5c91ed8207ba5856a8f7b108107b521a0c7e24cabf4bff182776afe"}
Sep 17 20:54:15.661573 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.661540    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:54:15.724211 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.724185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8vlp\" (UniqueName: \"kubernetes.io/projected/3afd5324-220d-42aa-8aa8-d424badc6efe-kube-api-access-m8vlp\") pod \"3afd5324-220d-42aa-8aa8-d424badc6efe\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") "
Sep 17 20:54:15.724418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.724222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-util\") pod \"3afd5324-220d-42aa-8aa8-d424badc6efe\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") "
Sep 17 20:54:15.724418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.724359    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-bundle\") pod \"3afd5324-220d-42aa-8aa8-d424badc6efe\" (UID: \"3afd5324-220d-42aa-8aa8-d424badc6efe\") "
Sep 17 20:54:15.725007 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.724926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-bundle" (OuterVolumeSpecName: "bundle") pod "3afd5324-220d-42aa-8aa8-d424badc6efe" (UID: "3afd5324-220d-42aa-8aa8-d424badc6efe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:54:15.726633 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.726606    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afd5324-220d-42aa-8aa8-d424badc6efe-kube-api-access-m8vlp" (OuterVolumeSpecName: "kube-api-access-m8vlp") pod "3afd5324-220d-42aa-8aa8-d424badc6efe" (UID: "3afd5324-220d-42aa-8aa8-d424badc6efe"). InnerVolumeSpecName "kube-api-access-m8vlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:54:15.729672 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.729646    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-util" (OuterVolumeSpecName: "util") pod "3afd5324-220d-42aa-8aa8-d424badc6efe" (UID: "3afd5324-220d-42aa-8aa8-d424badc6efe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:54:15.825711 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.825680    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:54:15.825711 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.825706    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vlp\" (UniqueName: \"kubernetes.io/projected/3afd5324-220d-42aa-8aa8-d424badc6efe-kube-api-access-m8vlp\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:54:15.825711 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:15.825715    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3afd5324-220d-42aa-8aa8-d424badc6efe-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:54:16.550791 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:16.550759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z" event={"ID":"3afd5324-220d-42aa-8aa8-d424badc6efe","Type":"ContainerDied","Data":"3f6e857bfc92340dce812ad2e83ad75074e1c8cfb5ac00d73f78119276bc2986"}
Sep 17 20:54:16.550791 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:16.550786    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avkl8z"
Sep 17 20:54:16.551224 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:16.550809    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6e857bfc92340dce812ad2e83ad75074e1c8cfb5ac00d73f78119276bc2986"
Sep 17 20:54:19.709923 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.709890    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"]
Sep 17 20:54:19.710282 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.710185    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerName="pull"
Sep 17 20:54:19.710282 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.710196    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerName="pull"
Sep 17 20:54:19.710282 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.710205    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerName="extract"
Sep 17 20:54:19.710282 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.710211    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerName="extract"
Sep 17 20:54:19.710282 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.710224    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerName="util"
Sep 17 20:54:19.710282 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.710230    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerName="util"
Sep 17 20:54:19.710282 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.710276    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afd5324-220d-42aa-8aa8-d424badc6efe" containerName="extract"
Sep 17 20:54:19.746838 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.746802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"]
Sep 17 20:54:19.746991 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.746894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:19.749529 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.749508    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 20:54:19.749643 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.749566    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 20:54:19.749986 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.749970    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 20:54:19.750031 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.749977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qlh4b\""
Sep 17 20:54:19.855408 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.855378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/546a2a05-e25f-4422-95e8-cab7733cfe36-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-246dt\" (UID: \"546a2a05-e25f-4422-95e8-cab7733cfe36\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:19.855576 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.855428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcww\" (UniqueName: \"kubernetes.io/projected/546a2a05-e25f-4422-95e8-cab7733cfe36-kube-api-access-4bcww\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-246dt\" (UID: \"546a2a05-e25f-4422-95e8-cab7733cfe36\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:19.956665 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.956628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcww\" (UniqueName: \"kubernetes.io/projected/546a2a05-e25f-4422-95e8-cab7733cfe36-kube-api-access-4bcww\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-246dt\" (UID: \"546a2a05-e25f-4422-95e8-cab7733cfe36\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:19.956839 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.956710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/546a2a05-e25f-4422-95e8-cab7733cfe36-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-246dt\" (UID: \"546a2a05-e25f-4422-95e8-cab7733cfe36\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:19.959158 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.959135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/546a2a05-e25f-4422-95e8-cab7733cfe36-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-246dt\" (UID: \"546a2a05-e25f-4422-95e8-cab7733cfe36\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:19.966347 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:19.966282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcww\" (UniqueName: \"kubernetes.io/projected/546a2a05-e25f-4422-95e8-cab7733cfe36-kube-api-access-4bcww\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-246dt\" (UID: \"546a2a05-e25f-4422-95e8-cab7733cfe36\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:20.057332 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:20.057293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:20.348904 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:20.348876    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"]
Sep 17 20:54:20.352158 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:54:20.352136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546a2a05_e25f_4422_95e8_cab7733cfe36.slice/crio-c004f63c2a611dd1996ea394b3e236abec53f64719f296d29acf1aaea8efaffb WatchSource:0}: Error finding container c004f63c2a611dd1996ea394b3e236abec53f64719f296d29acf1aaea8efaffb: Status 404 returned error can't find the container with id c004f63c2a611dd1996ea394b3e236abec53f64719f296d29acf1aaea8efaffb
Sep 17 20:54:20.572063 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:20.572028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt" event={"ID":"546a2a05-e25f-4422-95e8-cab7733cfe36","Type":"ContainerStarted","Data":"c004f63c2a611dd1996ea394b3e236abec53f64719f296d29acf1aaea8efaffb"}
Sep 17 20:54:22.208437 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:22.208407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 20:54:22.212059 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:22.212036    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 20:54:22.215635 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:22.215612    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:54:22.218521 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:22.218478    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:54:22.222369 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:22.222345    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 20:54:23.587381 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:23.587341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt" event={"ID":"546a2a05-e25f-4422-95e8-cab7733cfe36","Type":"ContainerStarted","Data":"e7aac91d909e5786866af59651a29f673760e72751ac5b1e92c6834dee7585bb"}
Sep 17 20:54:23.587735 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:23.587439    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:23.608338 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:23.608273    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt" podStartSLOduration=1.4837655029999999 podStartE2EDuration="4.608253855s" totalImagesPullingTime="3.124488352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:54:23.605782888 +0000 UTC m=+303.299592218" watchObservedRunningTime="2026-09-17 20:54:23.608253855 +0000 UTC m=+303.302063187"
Sep 17 20:54:24.210878 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.210843    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-pp8kb"]
Sep 17 20:54:24.214503 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.214485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.217145 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.217124    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 20:54:24.217314 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.217290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zkvfx\""
Sep 17 20:54:24.217449 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.217354    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 20:54:24.222658 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.222634    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-pp8kb"]
Sep 17 20:54:24.293127 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.293097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-certificates\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.293127 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.293127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdz9z\" (UniqueName: \"kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-kube-api-access-tdz9z\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.293363 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.293220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1443d232-0375-44ac-a951-5fe2c86d8139-cabundle0\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.394649 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.394609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1443d232-0375-44ac-a951-5fe2c86d8139-cabundle0\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.394649 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.394654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-certificates\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.394885 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.394671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdz9z\" (UniqueName: \"kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-kube-api-access-tdz9z\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.394885 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:54:24.394790    2579 secret.go:281] references non-existent secret key: ca.crt
Sep 17 20:54:24.394885 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:54:24.394814    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 20:54:24.394885 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:54:24.394826    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-pp8kb: references non-existent secret key: ca.crt
Sep 17 20:54:24.395075 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:54:24.394897    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-certificates podName:1443d232-0375-44ac-a951-5fe2c86d8139 nodeName:}" failed. No retries permitted until 2026-09-17 20:54:24.89487738 +0000 UTC m=+304.588686696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-certificates") pod "keda-operator-579c8958f7-pp8kb" (UID: "1443d232-0375-44ac-a951-5fe2c86d8139") : references non-existent secret key: ca.crt
Sep 17 20:54:24.395163 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.395140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1443d232-0375-44ac-a951-5fe2c86d8139-cabundle0\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.405281 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.405260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdz9z\" (UniqueName: \"kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-kube-api-access-tdz9z\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.899012 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.898982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-certificates\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.901647 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.901625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1443d232-0375-44ac-a951-5fe2c86d8139-certificates\") pod \"keda-operator-579c8958f7-pp8kb\" (UID: \"1443d232-0375-44ac-a951-5fe2c86d8139\") " pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:24.959905 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.959872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"]
Sep 17 20:54:24.963079 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.963063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:24.965119 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.965102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 20:54:24.976033 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:24.976014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"]
Sep 17 20:54:25.100677 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.100648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pjz\" (UniqueName: \"kubernetes.io/projected/9870030b-93a9-436e-a02c-381c53fba5ea-kube-api-access-l2pjz\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.100850 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.100696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9870030b-93a9-436e-a02c-381c53fba5ea-certificates\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.100850 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.100766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9870030b-93a9-436e-a02c-381c53fba5ea-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.125809 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.125775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:25.202204 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.202145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pjz\" (UniqueName: \"kubernetes.io/projected/9870030b-93a9-436e-a02c-381c53fba5ea-kube-api-access-l2pjz\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.202204 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.202202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9870030b-93a9-436e-a02c-381c53fba5ea-certificates\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.202651 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.202552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9870030b-93a9-436e-a02c-381c53fba5ea-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.202893 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.202872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9870030b-93a9-436e-a02c-381c53fba5ea-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.205962 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.205918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9870030b-93a9-436e-a02c-381c53fba5ea-certificates\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.211033 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.210920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pjz\" (UniqueName: \"kubernetes.io/projected/9870030b-93a9-436e-a02c-381c53fba5ea-kube-api-access-l2pjz\") pod \"keda-metrics-apiserver-77857b7b98-ptqh2\" (UID: \"9870030b-93a9-436e-a02c-381c53fba5ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.273300 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.273263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:25.406390 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.406355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-pp8kb"]
Sep 17 20:54:25.408569 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:54:25.408545    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1443d232_0375_44ac_a951_5fe2c86d8139.slice/crio-720514683ebabb81aa9c15cc5523bd83c80abcd4f22411f3d00fd10ae0ca7e66 WatchSource:0}: Error finding container 720514683ebabb81aa9c15cc5523bd83c80abcd4f22411f3d00fd10ae0ca7e66: Status 404 returned error can't find the container with id 720514683ebabb81aa9c15cc5523bd83c80abcd4f22411f3d00fd10ae0ca7e66
Sep 17 20:54:25.409834 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.409818    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:54:25.557945 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.557801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"]
Sep 17 20:54:25.559998 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:54:25.559973    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9870030b_93a9_436e_a02c_381c53fba5ea.slice/crio-aa8a8f09f7e8d5e58ce7db349d1122a668c9b91cfb331b8554a3603ea183ddae WatchSource:0}: Error finding container aa8a8f09f7e8d5e58ce7db349d1122a668c9b91cfb331b8554a3603ea183ddae: Status 404 returned error can't find the container with id aa8a8f09f7e8d5e58ce7db349d1122a668c9b91cfb331b8554a3603ea183ddae
Sep 17 20:54:25.597024 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.596990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2" event={"ID":"9870030b-93a9-436e-a02c-381c53fba5ea","Type":"ContainerStarted","Data":"aa8a8f09f7e8d5e58ce7db349d1122a668c9b91cfb331b8554a3603ea183ddae"}
Sep 17 20:54:25.598108 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:25.598089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-pp8kb" event={"ID":"1443d232-0375-44ac-a951-5fe2c86d8139","Type":"ContainerStarted","Data":"720514683ebabb81aa9c15cc5523bd83c80abcd4f22411f3d00fd10ae0ca7e66"}
Sep 17 20:54:30.623558 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:30.623521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-pp8kb" event={"ID":"1443d232-0375-44ac-a951-5fe2c86d8139","Type":"ContainerStarted","Data":"4997fdf69571455e49a0ea2dced11e744099233d7d07816edc05ade77e12a5b7"}
Sep 17 20:54:30.624084 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:30.623747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:54:30.625281 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:30.625247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2" event={"ID":"9870030b-93a9-436e-a02c-381c53fba5ea","Type":"ContainerStarted","Data":"b555a516c2784ee4d29bed12ddc1eb956311d8b1e276999d0189060589adfa33"}
Sep 17 20:54:30.625411 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:30.625383    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:30.641613 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:30.641569    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-pp8kb" podStartSLOduration=2.413553134 podStartE2EDuration="6.641554642s" totalImagesPullingTime="4.228001508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:54:30.638822222 +0000 UTC m=+310.332631552" watchObservedRunningTime="2026-09-17 20:54:30.641554642 +0000 UTC m=+310.335363950"
Sep 17 20:54:30.655510 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:30.655464    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2" podStartSLOduration=2.584703716 podStartE2EDuration="6.655446375s" totalImagesPullingTime="4.070742659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:54:30.654545785 +0000 UTC m=+310.348355135" watchObservedRunningTime="2026-09-17 20:54:30.655446375 +0000 UTC m=+310.349255711"
Sep 17 20:54:41.634779 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:41.634749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ptqh2"
Sep 17 20:54:44.594544 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:44.594516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-246dt"
Sep 17 20:54:51.632730 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:54:51.632702    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-pp8kb"
Sep 17 20:55:44.374218 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.374185    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-j6z92"]
Sep 17 20:55:44.377418 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.377399    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:44.381600 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.381558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 20:55:44.381709 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.381608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 20:55:44.381771 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.381713    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 20:55:44.382314 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.382299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qd5fb\""
Sep 17 20:55:44.391473 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.391450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-j6z92"]
Sep 17 20:55:44.514397 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.514369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2qq\" (UniqueName: \"kubernetes.io/projected/71705cd2-3866-4a19-8b27-17d13b03bfcd-kube-api-access-gd2qq\") pod \"kserve-controller-manager-66f4b8957f-j6z92\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") " pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:44.514397 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.514402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert\") pod \"kserve-controller-manager-66f4b8957f-j6z92\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") " pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:44.614887 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.614860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2qq\" (UniqueName: \"kubernetes.io/projected/71705cd2-3866-4a19-8b27-17d13b03bfcd-kube-api-access-gd2qq\") pod \"kserve-controller-manager-66f4b8957f-j6z92\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") " pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:44.615041 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.614892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert\") pod \"kserve-controller-manager-66f4b8957f-j6z92\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") " pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:44.615041 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:55:44.615017    2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 20:55:44.615109 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:55:44.615085    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert podName:71705cd2-3866-4a19-8b27-17d13b03bfcd nodeName:}" failed. No retries permitted until 2026-09-17 20:55:45.115066217 +0000 UTC m=+384.808875529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert") pod "kserve-controller-manager-66f4b8957f-j6z92" (UID: "71705cd2-3866-4a19-8b27-17d13b03bfcd") : secret "kserve-webhook-server-cert" not found
Sep 17 20:55:44.627491 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:44.627429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2qq\" (UniqueName: \"kubernetes.io/projected/71705cd2-3866-4a19-8b27-17d13b03bfcd-kube-api-access-gd2qq\") pod \"kserve-controller-manager-66f4b8957f-j6z92\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") " pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:45.119117 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:45.119084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert\") pod \"kserve-controller-manager-66f4b8957f-j6z92\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") " pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:45.121639 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:45.121616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert\") pod \"kserve-controller-manager-66f4b8957f-j6z92\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") " pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:45.288170 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:45.288140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:45.559412 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:45.559336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-j6z92"]
Sep 17 20:55:45.562510 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:55:45.562478    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71705cd2_3866_4a19_8b27_17d13b03bfcd.slice/crio-ffe48acaf3c41e96c820304d0b47b56c61b3611850f4aefc571777c81d20542b WatchSource:0}: Error finding container ffe48acaf3c41e96c820304d0b47b56c61b3611850f4aefc571777c81d20542b: Status 404 returned error can't find the container with id ffe48acaf3c41e96c820304d0b47b56c61b3611850f4aefc571777c81d20542b
Sep 17 20:55:45.945815 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:45.945777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92" event={"ID":"71705cd2-3866-4a19-8b27-17d13b03bfcd","Type":"ContainerStarted","Data":"ffe48acaf3c41e96c820304d0b47b56c61b3611850f4aefc571777c81d20542b"}
Sep 17 20:55:49.966622 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:49.966584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92" event={"ID":"71705cd2-3866-4a19-8b27-17d13b03bfcd","Type":"ContainerStarted","Data":"4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04"}
Sep 17 20:55:49.967035 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:49.966730    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:55:49.983614 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:49.983570    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92" podStartSLOduration=2.322946179 podStartE2EDuration="5.983558306s" totalImagesPullingTime="3.660612127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:49.982360527 +0000 UTC m=+389.676169878" watchObservedRunningTime="2026-09-17 20:55:49.983558306 +0000 UTC m=+389.677367671"
Sep 17 20:55:57.486989 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.486949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56475bb9fb-8bgs9"]
Sep 17 20:55:57.490796 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.490772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.501287 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.501259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56475bb9fb-8bgs9"]
Sep 17 20:55:57.621510 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.621474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-service-ca\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.621510 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.621515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-trusted-ca-bundle\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.621727 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.621541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c57be80-41fd-42b1-9014-38e10c64b125-console-oauth-config\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.621727 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.621620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c57be80-41fd-42b1-9014-38e10c64b125-console-serving-cert\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.621727 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.621653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxck\" (UniqueName: \"kubernetes.io/projected/6c57be80-41fd-42b1-9014-38e10c64b125-kube-api-access-wxxck\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.621727 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.621674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-console-config\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.621727 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.621703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-oauth-serving-cert\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723070 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-service-ca\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723070 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-trusted-ca-bundle\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723308 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c57be80-41fd-42b1-9014-38e10c64b125-console-oauth-config\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723308 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c57be80-41fd-42b1-9014-38e10c64b125-console-serving-cert\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723308 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxck\" (UniqueName: \"kubernetes.io/projected/6c57be80-41fd-42b1-9014-38e10c64b125-kube-api-access-wxxck\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723308 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-console-config\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723546 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-oauth-serving-cert\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.723977 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-service-ca\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.724078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.723988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-trusted-ca-bundle\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.724078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.724052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-oauth-serving-cert\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.724210 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.724187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c57be80-41fd-42b1-9014-38e10c64b125-console-config\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.725853 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.725826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c57be80-41fd-42b1-9014-38e10c64b125-console-oauth-config\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.725959 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.725944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c57be80-41fd-42b1-9014-38e10c64b125-console-serving-cert\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.731074 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.731056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxck\" (UniqueName: \"kubernetes.io/projected/6c57be80-41fd-42b1-9014-38e10c64b125-kube-api-access-wxxck\") pod \"console-56475bb9fb-8bgs9\" (UID: \"6c57be80-41fd-42b1-9014-38e10c64b125\") " pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:57.801145 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:57.801069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:55:58.074032 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:58.074005    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56475bb9fb-8bgs9"]
Sep 17 20:55:58.077736 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:55:58.077711    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c57be80_41fd_42b1_9014_38e10c64b125.slice/crio-9dd7455bebf1fc853173a6bc2a337dc0d6e233dfe0f46007fa4de26044d2a185 WatchSource:0}: Error finding container 9dd7455bebf1fc853173a6bc2a337dc0d6e233dfe0f46007fa4de26044d2a185: Status 404 returned error can't find the container with id 9dd7455bebf1fc853173a6bc2a337dc0d6e233dfe0f46007fa4de26044d2a185
Sep 17 20:55:59.005165 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:59.005131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56475bb9fb-8bgs9" event={"ID":"6c57be80-41fd-42b1-9014-38e10c64b125","Type":"ContainerStarted","Data":"99741f6cfabb78636530e0e5dbd6da1115944ea9bad0758b49b692d27bc19049"}
Sep 17 20:55:59.005627 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:59.005179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56475bb9fb-8bgs9" event={"ID":"6c57be80-41fd-42b1-9014-38e10c64b125","Type":"ContainerStarted","Data":"9dd7455bebf1fc853173a6bc2a337dc0d6e233dfe0f46007fa4de26044d2a185"}
Sep 17 20:55:59.024155 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:55:59.024099    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56475bb9fb-8bgs9" podStartSLOduration=2.024081648 podStartE2EDuration="2.024081648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:59.022725685 +0000 UTC m=+398.716535015" watchObservedRunningTime="2026-09-17 20:55:59.024081648 +0000 UTC m=+398.717890979"
Sep 17 20:56:07.801851 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:07.801817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:56:07.802349 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:07.801877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:56:07.806812 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:07.806791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:56:08.045744 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:08.045720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56475bb9fb-8bgs9"
Sep 17 20:56:08.102293 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:08.102262    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57948746bf-2tmcw"]
Sep 17 20:56:20.976736 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:20.976706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:56:27.230635 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.230597    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-j6z92"]
Sep 17 20:56:27.231064 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.230782    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92" podUID="71705cd2-3866-4a19-8b27-17d13b03bfcd" containerName="manager" containerID="cri-o://4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04" gracePeriod=10
Sep 17 20:56:27.259778 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.259755    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-b4kdd"]
Sep 17 20:56:27.263056 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.263041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.275307 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.275286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-b4kdd"]
Sep 17 20:56:27.366554 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.366521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5cqw\" (UniqueName: \"kubernetes.io/projected/3b46c6e2-cec1-412f-81ef-30252529a6bf-kube-api-access-d5cqw\") pod \"kserve-controller-manager-66f4b8957f-b4kdd\" (UID: \"3b46c6e2-cec1-412f-81ef-30252529a6bf\") " pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.366740 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.366563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b46c6e2-cec1-412f-81ef-30252529a6bf-cert\") pod \"kserve-controller-manager-66f4b8957f-b4kdd\" (UID: \"3b46c6e2-cec1-412f-81ef-30252529a6bf\") " pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.465440 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.465414    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:56:27.467489 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.467469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cqw\" (UniqueName: \"kubernetes.io/projected/3b46c6e2-cec1-412f-81ef-30252529a6bf-kube-api-access-d5cqw\") pod \"kserve-controller-manager-66f4b8957f-b4kdd\" (UID: \"3b46c6e2-cec1-412f-81ef-30252529a6bf\") " pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.467550 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.467504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b46c6e2-cec1-412f-81ef-30252529a6bf-cert\") pod \"kserve-controller-manager-66f4b8957f-b4kdd\" (UID: \"3b46c6e2-cec1-412f-81ef-30252529a6bf\") " pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.469785 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.469762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b46c6e2-cec1-412f-81ef-30252529a6bf-cert\") pod \"kserve-controller-manager-66f4b8957f-b4kdd\" (UID: \"3b46c6e2-cec1-412f-81ef-30252529a6bf\") " pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.476150 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.476131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cqw\" (UniqueName: \"kubernetes.io/projected/3b46c6e2-cec1-412f-81ef-30252529a6bf-kube-api-access-d5cqw\") pod \"kserve-controller-manager-66f4b8957f-b4kdd\" (UID: \"3b46c6e2-cec1-412f-81ef-30252529a6bf\") " pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.568664 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.568576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert\") pod \"71705cd2-3866-4a19-8b27-17d13b03bfcd\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") "
Sep 17 20:56:27.568816 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.568678    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd2qq\" (UniqueName: \"kubernetes.io/projected/71705cd2-3866-4a19-8b27-17d13b03bfcd-kube-api-access-gd2qq\") pod \"71705cd2-3866-4a19-8b27-17d13b03bfcd\" (UID: \"71705cd2-3866-4a19-8b27-17d13b03bfcd\") "
Sep 17 20:56:27.570791 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.570762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert" (OuterVolumeSpecName: "cert") pod "71705cd2-3866-4a19-8b27-17d13b03bfcd" (UID: "71705cd2-3866-4a19-8b27-17d13b03bfcd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:56:27.570900 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.570792    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71705cd2-3866-4a19-8b27-17d13b03bfcd-kube-api-access-gd2qq" (OuterVolumeSpecName: "kube-api-access-gd2qq") pod "71705cd2-3866-4a19-8b27-17d13b03bfcd" (UID: "71705cd2-3866-4a19-8b27-17d13b03bfcd"). InnerVolumeSpecName "kube-api-access-gd2qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:56:27.628088 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.628065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:27.669412 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.669383    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2qq\" (UniqueName: \"kubernetes.io/projected/71705cd2-3866-4a19-8b27-17d13b03bfcd-kube-api-access-gd2qq\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:27.669412 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.669411    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71705cd2-3866-4a19-8b27-17d13b03bfcd-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:27.897826 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:27.897765    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-b4kdd"]
Sep 17 20:56:27.899878 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:56:27.899851    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b46c6e2_cec1_412f_81ef_30252529a6bf.slice/crio-a6128cd9856a19fce8a1ca02c04a8445c559c6b2c23531f08bae7c66c69fd345 WatchSource:0}: Error finding container a6128cd9856a19fce8a1ca02c04a8445c559c6b2c23531f08bae7c66c69fd345: Status 404 returned error can't find the container with id a6128cd9856a19fce8a1ca02c04a8445c559c6b2c23531f08bae7c66c69fd345
Sep 17 20:56:28.127824 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.127787    2579 generic.go:356] "Generic (PLEG): container finished" podID="71705cd2-3866-4a19-8b27-17d13b03bfcd" containerID="4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04" exitCode=0
Sep 17 20:56:28.127824 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.127818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92" event={"ID":"71705cd2-3866-4a19-8b27-17d13b03bfcd","Type":"ContainerDied","Data":"4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04"}
Sep 17 20:56:28.128078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.127870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92" event={"ID":"71705cd2-3866-4a19-8b27-17d13b03bfcd","Type":"ContainerDied","Data":"ffe48acaf3c41e96c820304d0b47b56c61b3611850f4aefc571777c81d20542b"}
Sep 17 20:56:28.128078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.127828    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-j6z92"
Sep 17 20:56:28.128078 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.127899    2579 scope.go:117] "RemoveContainer" containerID="4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04"
Sep 17 20:56:28.129265 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.129236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd" event={"ID":"3b46c6e2-cec1-412f-81ef-30252529a6bf","Type":"ContainerStarted","Data":"a6128cd9856a19fce8a1ca02c04a8445c559c6b2c23531f08bae7c66c69fd345"}
Sep 17 20:56:28.136797 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.136780    2579 scope.go:117] "RemoveContainer" containerID="4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04"
Sep 17 20:56:28.137069 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:56:28.137047    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04\": container with ID starting with 4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04 not found: ID does not exist" containerID="4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04"
Sep 17 20:56:28.137130 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.137081    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04"} err="failed to get container status \"4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04\": rpc error: code = NotFound desc = could not find container \"4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04\": container with ID starting with 4a2abe43838c9a6b9477f878e71e6ea26fe89d314b9c9f535b310edea8353b04 not found: ID does not exist"
Sep 17 20:56:28.158794 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.158742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-j6z92"]
Sep 17 20:56:28.161821 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.161804    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-j6z92"]
Sep 17 20:56:28.300548 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:28.300524    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71705cd2-3866-4a19-8b27-17d13b03bfcd" path="/var/lib/kubelet/pods/71705cd2-3866-4a19-8b27-17d13b03bfcd/volumes"
Sep 17 20:56:29.135572 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:29.135536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd" event={"ID":"3b46c6e2-cec1-412f-81ef-30252529a6bf","Type":"ContainerStarted","Data":"57dfaae3a99312669d5d0749d5d18ac4f365c3846e9704f9b81a967b09321e36"}
Sep 17 20:56:29.135743 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:29.135592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:56:29.155082 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:29.155032    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd" podStartSLOduration=1.777563145 podStartE2EDuration="2.155015381s" totalImagesPullingTime="377.452236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:29.15381239 +0000 UTC m=+428.847621722" watchObservedRunningTime="2026-09-17 20:56:29.155015381 +0000 UTC m=+428.848824711"
Sep 17 20:56:33.122142 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.122056    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57948746bf-2tmcw" podUID="dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" containerName="console" containerID="cri-o://595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5" gracePeriod=15
Sep 17 20:56:33.370807 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.370786    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:56:33.518841 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.518754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-oauth-config\") pod \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") "
Sep 17 20:56:33.518841 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.518793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsxc5\" (UniqueName: \"kubernetes.io/projected/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-kube-api-access-zsxc5\") pod \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") "
Sep 17 20:56:33.518841 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.518810    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-service-ca\") pod \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") "
Sep 17 20:56:33.518841 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.518830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-config\") pod \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") "
Sep 17 20:56:33.518841 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.518845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-trusted-ca-bundle\") pod \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") "
Sep 17 20:56:33.519207 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.518895    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-oauth-serving-cert\") pod \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") "
Sep 17 20:56:33.519207 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.518915    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-serving-cert\") pod \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\" (UID: \"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0\") "
Sep 17 20:56:33.519348 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.519285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-config" (OuterVolumeSpecName: "console-config") pod "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" (UID: "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:56:33.519424 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.519362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-service-ca" (OuterVolumeSpecName: "service-ca") pod "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" (UID: "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:56:33.519424 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.519378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" (UID: "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:56:33.519496 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.519447    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" (UID: "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:56:33.521157 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.521125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-kube-api-access-zsxc5" (OuterVolumeSpecName: "kube-api-access-zsxc5") pod "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" (UID: "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0"). InnerVolumeSpecName "kube-api-access-zsxc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:56:33.521266 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.521196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" (UID: "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:56:33.521266 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.521223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" (UID: "dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:56:33.620114 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.620082    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-oauth-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:33.620114 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.620107    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxc5\" (UniqueName: \"kubernetes.io/projected/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-kube-api-access-zsxc5\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:33.620114 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.620118    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:33.620356 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.620127    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:33.620356 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.620135    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-trusted-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:33.620356 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.620143    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-oauth-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:33.620356 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:33.620151    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0-console-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 20:56:34.157547 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.157509    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" containerID="595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5" exitCode=0
Sep 17 20:56:34.157924 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.157559    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57948746bf-2tmcw"
Sep 17 20:56:34.157924 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.157570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57948746bf-2tmcw" event={"ID":"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0","Type":"ContainerDied","Data":"595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5"}
Sep 17 20:56:34.157924 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.157629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57948746bf-2tmcw" event={"ID":"dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0","Type":"ContainerDied","Data":"5f8daf12e0a10e63b2947915f55cb009ff39eabe67e4232868fc9f31656536b4"}
Sep 17 20:56:34.157924 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.157656    2579 scope.go:117] "RemoveContainer" containerID="595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5"
Sep 17 20:56:34.166554 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.166537    2579 scope.go:117] "RemoveContainer" containerID="595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5"
Sep 17 20:56:34.166819 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:56:34.166800    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5\": container with ID starting with 595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5 not found: ID does not exist" containerID="595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5"
Sep 17 20:56:34.166880 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.166831    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5"} err="failed to get container status \"595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5\": rpc error: code = NotFound desc = could not find container \"595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5\": container with ID starting with 595c0d25d188fd706a29d56b5afc83f32b59e10fbe5fc2fe65b2b800e53e9ad5 not found: ID does not exist"
Sep 17 20:56:34.184305 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.184280    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57948746bf-2tmcw"]
Sep 17 20:56:34.189998 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.189978    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57948746bf-2tmcw"]
Sep 17 20:56:34.298981 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:56:34.298949    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" path="/var/lib/kubelet/pods/dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0/volumes"
Sep 17 20:57:00.145526 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:00.145488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f4b8957f-b4kdd"
Sep 17 20:57:01.415549 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.415514    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-fnlkh"]
Sep 17 20:57:01.415998 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.415982    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" containerName="console"
Sep 17 20:57:01.416060 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.416001    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" containerName="console"
Sep 17 20:57:01.416060 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.416028    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71705cd2-3866-4a19-8b27-17d13b03bfcd" containerName="manager"
Sep 17 20:57:01.416060 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.416038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71705cd2-3866-4a19-8b27-17d13b03bfcd" containerName="manager"
Sep 17 20:57:01.416198 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.416131    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbccfb2e-98b4-4c5f-8bc2-ed7c4d293ed0" containerName="console"
Sep 17 20:57:01.416198 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.416148    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71705cd2-3866-4a19-8b27-17d13b03bfcd" containerName="manager"
Sep 17 20:57:01.420925 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.420904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:01.423665 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.423645    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-ldwdq\""
Sep 17 20:57:01.423963 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.423687    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 20:57:01.425797 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.425736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b96f7b-158c-473a-ae3a-4b46471f62d2-tls-certs\") pod \"model-serving-api-7b954ffc95-fnlkh\" (UID: \"b9b96f7b-158c-473a-ae3a-4b46471f62d2\") " pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:01.425915 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.425837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sssx8\" (UniqueName: \"kubernetes.io/projected/b9b96f7b-158c-473a-ae3a-4b46471f62d2-kube-api-access-sssx8\") pod \"model-serving-api-7b954ffc95-fnlkh\" (UID: \"b9b96f7b-158c-473a-ae3a-4b46471f62d2\") " pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:01.433612 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.433590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-fnlkh"]
Sep 17 20:57:01.526814 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.526776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sssx8\" (UniqueName: \"kubernetes.io/projected/b9b96f7b-158c-473a-ae3a-4b46471f62d2-kube-api-access-sssx8\") pod \"model-serving-api-7b954ffc95-fnlkh\" (UID: \"b9b96f7b-158c-473a-ae3a-4b46471f62d2\") " pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:01.527005 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.526863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b96f7b-158c-473a-ae3a-4b46471f62d2-tls-certs\") pod \"model-serving-api-7b954ffc95-fnlkh\" (UID: \"b9b96f7b-158c-473a-ae3a-4b46471f62d2\") " pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:01.527005 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:57:01.526946    2579 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 20:57:01.527124 ip-10-0-139-55 kubenswrapper[2579]: E0917 20:57:01.527028    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9b96f7b-158c-473a-ae3a-4b46471f62d2-tls-certs podName:b9b96f7b-158c-473a-ae3a-4b46471f62d2 nodeName:}" failed. No retries permitted until 2026-09-17 20:57:02.027007799 +0000 UTC m=+461.720817120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b9b96f7b-158c-473a-ae3a-4b46471f62d2-tls-certs") pod "model-serving-api-7b954ffc95-fnlkh" (UID: "b9b96f7b-158c-473a-ae3a-4b46471f62d2") : secret "model-serving-api-tls" not found
Sep 17 20:57:01.535731 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:01.535705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sssx8\" (UniqueName: \"kubernetes.io/projected/b9b96f7b-158c-473a-ae3a-4b46471f62d2-kube-api-access-sssx8\") pod \"model-serving-api-7b954ffc95-fnlkh\" (UID: \"b9b96f7b-158c-473a-ae3a-4b46471f62d2\") " pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:02.029618 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:02.029583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b96f7b-158c-473a-ae3a-4b46471f62d2-tls-certs\") pod \"model-serving-api-7b954ffc95-fnlkh\" (UID: \"b9b96f7b-158c-473a-ae3a-4b46471f62d2\") " pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:02.032029 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:02.032011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b96f7b-158c-473a-ae3a-4b46471f62d2-tls-certs\") pod \"model-serving-api-7b954ffc95-fnlkh\" (UID: \"b9b96f7b-158c-473a-ae3a-4b46471f62d2\") " pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:02.033972 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:02.033949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:02.305053 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:02.304988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-fnlkh"]
Sep 17 20:57:02.307338 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:57:02.307296    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b96f7b_158c_473a_ae3a_4b46471f62d2.slice/crio-ce399f4036065f1c04051c3dc75a4673905324b387150f76100711680727e88d WatchSource:0}: Error finding container ce399f4036065f1c04051c3dc75a4673905324b387150f76100711680727e88d: Status 404 returned error can't find the container with id ce399f4036065f1c04051c3dc75a4673905324b387150f76100711680727e88d
Sep 17 20:57:03.287447 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:03.287404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-fnlkh" event={"ID":"b9b96f7b-158c-473a-ae3a-4b46471f62d2","Type":"ContainerStarted","Data":"ce399f4036065f1c04051c3dc75a4673905324b387150f76100711680727e88d"}
Sep 17 20:57:05.298624 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:05.298594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-fnlkh" event={"ID":"b9b96f7b-158c-473a-ae3a-4b46471f62d2","Type":"ContainerStarted","Data":"8173730453e7e04cad312fc2ee4b2f982f39866780d5390723419c8e57239408"}
Sep 17 20:57:05.298992 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:05.298775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:57:05.320447 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:05.320399    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-fnlkh" podStartSLOduration=2.267447587 podStartE2EDuration="4.320384747s" totalImagesPullingTime="2.05293716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:05.318885435 +0000 UTC m=+465.012694765" watchObservedRunningTime="2026-09-17 20:57:05.320384747 +0000 UTC m=+465.014194085"
Sep 17 20:57:16.306225 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:57:16.306195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-fnlkh"
Sep 17 20:58:48.476627 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.476579    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"]
Sep 17 20:58:48.479928 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.479912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.484868 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.484842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a625d-kube-rbac-proxy-sar-config\""
Sep 17 20:58:48.485002 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.484938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a625d-predictor-serving-cert\""
Sep 17 20:58:48.485822 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.485798    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 20:58:48.485944 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.485811    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tx4r4\""
Sep 17 20:58:48.485944 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.485920    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 20:58:48.501699 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.501675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"]
Sep 17 20:58:48.584998 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.584964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2bd31b-92e4-47d4-8109-ecec6131c122-proxy-tls\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.585261 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.585234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.585414 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.585359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cv4s\" (UniqueName: \"kubernetes.io/projected/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-api-access-9cv4s\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.686821 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.686786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cv4s\" (UniqueName: \"kubernetes.io/projected/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-api-access-9cv4s\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.686979 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.686857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2bd31b-92e4-47d4-8109-ecec6131c122-proxy-tls\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.686979 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.686898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.687712 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.687691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.689601 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.689574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2bd31b-92e4-47d4-8109-ecec6131c122-proxy-tls\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.696229 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.696202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cv4s\" (UniqueName: \"kubernetes.io/projected/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-api-access-9cv4s\") pod \"success-200-isvc-a625d-predictor-68cf5d77c9-k7djz\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") " pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:48.790665 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:48.790577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:58:49.118469 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:49.118438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"]
Sep 17 20:58:49.121866 ip-10-0-139-55 kubenswrapper[2579]: W0917 20:58:49.121842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2bd31b_92e4_47d4_8109_ecec6131c122.slice/crio-a876632cf2896e4ed5afbf3b28647b2c3b4e31ea17f234383e12ad6bf14e63bd WatchSource:0}: Error finding container a876632cf2896e4ed5afbf3b28647b2c3b4e31ea17f234383e12ad6bf14e63bd: Status 404 returned error can't find the container with id a876632cf2896e4ed5afbf3b28647b2c3b4e31ea17f234383e12ad6bf14e63bd
Sep 17 20:58:49.752315 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:58:49.752279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" event={"ID":"be2bd31b-92e4-47d4-8109-ecec6131c122","Type":"ContainerStarted","Data":"a876632cf2896e4ed5afbf3b28647b2c3b4e31ea17f234383e12ad6bf14e63bd"}
Sep 17 20:59:01.827924 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:01.827884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" event={"ID":"be2bd31b-92e4-47d4-8109-ecec6131c122","Type":"ContainerStarted","Data":"a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90"}
Sep 17 20:59:03.839931 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:03.839896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" event={"ID":"be2bd31b-92e4-47d4-8109-ecec6131c122","Type":"ContainerStarted","Data":"dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494"}
Sep 17 20:59:03.840437 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:03.839974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:59:03.840437 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:03.840097    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:59:03.841669 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:03.841641    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 20:59:03.861646 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:03.861596    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podStartSLOduration=1.693613247 podStartE2EDuration="15.861576571s" totalImagesPullingTime="14.167963324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:03.860399585 +0000 UTC m=+583.554208936" watchObservedRunningTime="2026-09-17 20:59:03.861576571 +0000 UTC m=+583.555385901"
Sep 17 20:59:04.844263 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:04.844222    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 20:59:09.849047 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:09.849018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 20:59:09.849667 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:09.849635    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 20:59:19.850538 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:19.850493    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 20:59:22.245186 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:22.245152    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 20:59:22.245648 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:22.245408    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 20:59:22.249653 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:22.249635    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:59:22.249780 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:22.249681    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 20:59:29.849820 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:29.849777    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 20:59:39.850607 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:39.850521    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 20:59:49.850785 ip-10-0-139-55 kubenswrapper[2579]: I0917 20:59:49.850756    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 21:00:08.148648 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.148618    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"]
Sep 17 21:00:08.151803 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.151788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:08.154057 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.154022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a625d-serving-cert\""
Sep 17 21:00:08.154187 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.154072    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a625d-kube-rbac-proxy-sar-config\""
Sep 17 21:00:08.159913 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.159889    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"]
Sep 17 21:00:08.256745 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.256712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls\") pod \"switch-graph-a625d-77f5ffd85c-nwwfg\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") " pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:08.256900 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.256763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211f2575-61fb-439a-a3d9-c32ae4c6b79b-openshift-service-ca-bundle\") pod \"switch-graph-a625d-77f5ffd85c-nwwfg\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") " pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:08.358137 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.358111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls\") pod \"switch-graph-a625d-77f5ffd85c-nwwfg\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") " pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:08.358347 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.358145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211f2575-61fb-439a-a3d9-c32ae4c6b79b-openshift-service-ca-bundle\") pod \"switch-graph-a625d-77f5ffd85c-nwwfg\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") " pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:08.358347 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:00:08.358312    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-a625d-serving-cert: secret "switch-graph-a625d-serving-cert" not found
Sep 17 21:00:08.358471 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:00:08.358407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls podName:211f2575-61fb-439a-a3d9-c32ae4c6b79b nodeName:}" failed. No retries permitted until 2026-09-17 21:00:08.858391083 +0000 UTC m=+648.552200393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls") pod "switch-graph-a625d-77f5ffd85c-nwwfg" (UID: "211f2575-61fb-439a-a3d9-c32ae4c6b79b") : secret "switch-graph-a625d-serving-cert" not found
Sep 17 21:00:08.358862 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.358841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211f2575-61fb-439a-a3d9-c32ae4c6b79b-openshift-service-ca-bundle\") pod \"switch-graph-a625d-77f5ffd85c-nwwfg\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") " pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:08.863288 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.863255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls\") pod \"switch-graph-a625d-77f5ffd85c-nwwfg\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") " pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:08.865915 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:08.865890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls\") pod \"switch-graph-a625d-77f5ffd85c-nwwfg\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") " pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:09.062981 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:09.062949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:09.338485 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:09.338460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"]
Sep 17 21:00:09.341207 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:00:09.341176    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211f2575_61fb_439a_a3d9_c32ae4c6b79b.slice/crio-c971932c535924aa5edadab8adade3d95d0da5cd60bbd17c2c7c57cdf1a75494 WatchSource:0}: Error finding container c971932c535924aa5edadab8adade3d95d0da5cd60bbd17c2c7c57cdf1a75494: Status 404 returned error can't find the container with id c971932c535924aa5edadab8adade3d95d0da5cd60bbd17c2c7c57cdf1a75494
Sep 17 21:00:09.343240 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:09.343224    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:00:10.124949 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:10.124908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" event={"ID":"211f2575-61fb-439a-a3d9-c32ae4c6b79b","Type":"ContainerStarted","Data":"c971932c535924aa5edadab8adade3d95d0da5cd60bbd17c2c7c57cdf1a75494"}
Sep 17 21:00:12.134881 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:12.134847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" event={"ID":"211f2575-61fb-439a-a3d9-c32ae4c6b79b","Type":"ContainerStarted","Data":"40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b"}
Sep 17 21:00:12.135285 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:12.134992    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:12.155144 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:12.155104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podStartSLOduration=2.129735775 podStartE2EDuration="4.155090481s" totalImagesPullingTime="2.025354706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:12.154604922 +0000 UTC m=+651.848414253" watchObservedRunningTime="2026-09-17 21:00:12.155090481 +0000 UTC m=+651.848899812"
Sep 17 21:00:18.145462 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:18.145431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:23.155886 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.155848    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"]
Sep 17 21:00:23.156421 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.156085    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d" containerID="cri-o://40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b" gracePeriod=30
Sep 17 21:00:23.293936 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.293897    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"]
Sep 17 21:00:23.294161 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.294137    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container" containerID="cri-o://a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90" gracePeriod=30
Sep 17 21:00:23.294256 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.294191    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kube-rbac-proxy" containerID="cri-o://dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494" gracePeriod=30
Sep 17 21:00:23.445717 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.445687    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"]
Sep 17 21:00:23.450965 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.450588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.453048 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.453022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-23363-predictor-serving-cert\""
Sep 17 21:00:23.453180 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.453023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-23363-kube-rbac-proxy-sar-config\""
Sep 17 21:00:23.497998 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.497965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"]
Sep 17 21:00:23.587896 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.587868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.588079 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.587905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e17d71-83fc-41f5-aee2-a16cf61edb83-proxy-tls\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.588079 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.587928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjst\" (UniqueName: \"kubernetes.io/projected/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-api-access-jvjst\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.689404 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.689304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.689404 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.689354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e17d71-83fc-41f5-aee2-a16cf61edb83-proxy-tls\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.689404 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.689376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjst\" (UniqueName: \"kubernetes.io/projected/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-api-access-jvjst\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.690064 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.690042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.692085 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.692060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e17d71-83fc-41f5-aee2-a16cf61edb83-proxy-tls\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.699926 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.699900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjst\" (UniqueName: \"kubernetes.io/projected/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-api-access-jvjst\") pod \"success-200-isvc-23363-predictor-5875556545-7gqbn\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") " pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:23.763116 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:23.763087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:24.036689 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.036615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"]
Sep 17 21:00:24.039434 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:00:24.039407    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e17d71_83fc_41f5_aee2_a16cf61edb83.slice/crio-170a67a7979b3d906733b2170f2221c34ecbe66835569728220547449becdc98 WatchSource:0}: Error finding container 170a67a7979b3d906733b2170f2221c34ecbe66835569728220547449becdc98: Status 404 returned error can't find the container with id 170a67a7979b3d906733b2170f2221c34ecbe66835569728220547449becdc98
Sep 17 21:00:24.191632 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.191588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" event={"ID":"25e17d71-83fc-41f5-aee2-a16cf61edb83","Type":"ContainerStarted","Data":"3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff"}
Sep 17 21:00:24.192343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.191846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:24.192343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.191903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" event={"ID":"25e17d71-83fc-41f5-aee2-a16cf61edb83","Type":"ContainerStarted","Data":"ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db"}
Sep 17 21:00:24.192343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.191938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" event={"ID":"25e17d71-83fc-41f5-aee2-a16cf61edb83","Type":"ContainerStarted","Data":"170a67a7979b3d906733b2170f2221c34ecbe66835569728220547449becdc98"}
Sep 17 21:00:24.192343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.191987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:24.194125 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.194095    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:00:24.194234 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.194139    2579 generic.go:356] "Generic (PLEG): container finished" podID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerID="dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494" exitCode=2
Sep 17 21:00:24.194234 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.194210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" event={"ID":"be2bd31b-92e4-47d4-8109-ecec6131c122","Type":"ContainerDied","Data":"dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494"}
Sep 17 21:00:24.209063 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.209022    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podStartSLOduration=1.209010198 podStartE2EDuration="1.209010198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:24.207594201 +0000 UTC m=+663.901403532" watchObservedRunningTime="2026-09-17 21:00:24.209010198 +0000 UTC m=+663.902819588"
Sep 17 21:00:24.845154 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:24.845119    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 21:00:25.198568 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:25.198484    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:00:26.660919 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.660892    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 21:00:26.818057 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.817961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cv4s\" (UniqueName: \"kubernetes.io/projected/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-api-access-9cv4s\") pod \"be2bd31b-92e4-47d4-8109-ecec6131c122\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") "
Sep 17 21:00:26.818057 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.818010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-rbac-proxy-sar-config\") pod \"be2bd31b-92e4-47d4-8109-ecec6131c122\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") "
Sep 17 21:00:26.818279 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.818190    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2bd31b-92e4-47d4-8109-ecec6131c122-proxy-tls\") pod \"be2bd31b-92e4-47d4-8109-ecec6131c122\" (UID: \"be2bd31b-92e4-47d4-8109-ecec6131c122\") "
Sep 17 21:00:26.818420 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.818397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be2bd31b-92e4-47d4-8109-ecec6131c122" (UID: "be2bd31b-92e4-47d4-8109-ecec6131c122"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:00:26.820350 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.820306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2bd31b-92e4-47d4-8109-ecec6131c122-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be2bd31b-92e4-47d4-8109-ecec6131c122" (UID: "be2bd31b-92e4-47d4-8109-ecec6131c122"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:00:26.820464 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.820380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-api-access-9cv4s" (OuterVolumeSpecName: "kube-api-access-9cv4s") pod "be2bd31b-92e4-47d4-8109-ecec6131c122" (UID: "be2bd31b-92e4-47d4-8109-ecec6131c122"). InnerVolumeSpecName "kube-api-access-9cv4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:26.918834 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.918808    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:00:26.918834 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.918832    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2bd31b-92e4-47d4-8109-ecec6131c122-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:00:26.919021 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:26.918842    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cv4s\" (UniqueName: \"kubernetes.io/projected/be2bd31b-92e4-47d4-8109-ecec6131c122-kube-api-access-9cv4s\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:00:27.209719 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.209688    2579 generic.go:356] "Generic (PLEG): container finished" podID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerID="a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90" exitCode=0
Sep 17 21:00:27.209904 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.209742    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"
Sep 17 21:00:27.209904 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.209771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" event={"ID":"be2bd31b-92e4-47d4-8109-ecec6131c122","Type":"ContainerDied","Data":"a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90"}
Sep 17 21:00:27.209904 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.209830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz" event={"ID":"be2bd31b-92e4-47d4-8109-ecec6131c122","Type":"ContainerDied","Data":"a876632cf2896e4ed5afbf3b28647b2c3b4e31ea17f234383e12ad6bf14e63bd"}
Sep 17 21:00:27.209904 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.209857    2579 scope.go:117] "RemoveContainer" containerID="dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494"
Sep 17 21:00:27.220817 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.220798    2579 scope.go:117] "RemoveContainer" containerID="a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90"
Sep 17 21:00:27.228395 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.228380    2579 scope.go:117] "RemoveContainer" containerID="dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494"
Sep 17 21:00:27.228636 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:00:27.228620    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494\": container with ID starting with dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494 not found: ID does not exist" containerID="dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494"
Sep 17 21:00:27.228706 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.228646    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494"} err="failed to get container status \"dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494\": rpc error: code = NotFound desc = could not find container \"dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494\": container with ID starting with dd29c89630f71074b70989a9592f90b35bdf2734825e7f7493b1cfe767e20494 not found: ID does not exist"
Sep 17 21:00:27.228706 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.228662    2579 scope.go:117] "RemoveContainer" containerID="a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90"
Sep 17 21:00:27.228904 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:00:27.228888    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90\": container with ID starting with a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90 not found: ID does not exist" containerID="a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90"
Sep 17 21:00:27.228939 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.228909    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90"} err="failed to get container status \"a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90\": rpc error: code = NotFound desc = could not find container \"a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90\": container with ID starting with a4c2b9a6a27fbd4b00111e584e0709d9eedc28772596373d97bb948768652f90 not found: ID does not exist"
Sep 17 21:00:27.235436 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.235415    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"]
Sep 17 21:00:27.241541 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:27.241519    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz"]
Sep 17 21:00:28.143987 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:28.143952    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:00:28.299549 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:28.299520    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" path="/var/lib/kubelet/pods/be2bd31b-92e4-47d4-8109-ecec6131c122/volumes"
Sep 17 21:00:30.202771 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:30.202747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:00:30.203286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:30.203258    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:00:33.143232 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:33.143185    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:00:38.143597 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:38.143560    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:00:38.144010 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:38.143673    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:40.203212 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:40.203172    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:00:43.143834 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:43.143795    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:00:48.143998 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:48.143958    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:00:50.203227 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:50.203186    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:00:53.143591 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.143544    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:00:53.307603 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.307580    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:53.325347 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.325293    2579 generic.go:356] "Generic (PLEG): container finished" podID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerID="40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b" exitCode=0
Sep 17 21:00:53.325484 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.325370    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"
Sep 17 21:00:53.325484 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.325377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" event={"ID":"211f2575-61fb-439a-a3d9-c32ae4c6b79b","Type":"ContainerDied","Data":"40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b"}
Sep 17 21:00:53.325484 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.325427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg" event={"ID":"211f2575-61fb-439a-a3d9-c32ae4c6b79b","Type":"ContainerDied","Data":"c971932c535924aa5edadab8adade3d95d0da5cd60bbd17c2c7c57cdf1a75494"}
Sep 17 21:00:53.325484 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.325450    2579 scope.go:117] "RemoveContainer" containerID="40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b"
Sep 17 21:00:53.333808 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.333783    2579 scope.go:117] "RemoveContainer" containerID="40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b"
Sep 17 21:00:53.334114 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:00:53.334089    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b\": container with ID starting with 40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b not found: ID does not exist" containerID="40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b"
Sep 17 21:00:53.334208 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.334119    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b"} err="failed to get container status \"40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b\": rpc error: code = NotFound desc = could not find container \"40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b\": container with ID starting with 40d2b5760cb80e989abdc9cb008a1730814a634d083772b9137ccf1fa6fe477b not found: ID does not exist"
Sep 17 21:00:53.441118 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.441024    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211f2575-61fb-439a-a3d9-c32ae4c6b79b-openshift-service-ca-bundle\") pod \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") "
Sep 17 21:00:53.441118 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.441112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls\") pod \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\" (UID: \"211f2575-61fb-439a-a3d9-c32ae4c6b79b\") "
Sep 17 21:00:53.441423 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.441402    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211f2575-61fb-439a-a3d9-c32ae4c6b79b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "211f2575-61fb-439a-a3d9-c32ae4c6b79b" (UID: "211f2575-61fb-439a-a3d9-c32ae4c6b79b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:00:53.443292 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.443264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "211f2575-61fb-439a-a3d9-c32ae4c6b79b" (UID: "211f2575-61fb-439a-a3d9-c32ae4c6b79b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:00:53.542525 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.542485    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211f2575-61fb-439a-a3d9-c32ae4c6b79b-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:00:53.542525 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.542518    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f2575-61fb-439a-a3d9-c32ae4c6b79b-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:00:53.648678 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.648646    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"]
Sep 17 21:00:53.652136 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:53.652111    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg"]
Sep 17 21:00:54.298951 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:00:54.298920    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" path="/var/lib/kubelet/pods/211f2575-61fb-439a-a3d9-c32ae4c6b79b/volumes"
Sep 17 21:01:00.203625 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:00.203541    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:01:08.240022 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.239950    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"]
Sep 17 21:01:08.240445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container"
Sep 17 21:01:08.240445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240374    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container"
Sep 17 21:01:08.240445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240400    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d"
Sep 17 21:01:08.240445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240407    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d"
Sep 17 21:01:08.240445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240417    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kube-rbac-proxy"
Sep 17 21:01:08.240445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240423    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kube-rbac-proxy"
Sep 17 21:01:08.240635 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240476    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="211f2575-61fb-439a-a3d9-c32ae4c6b79b" containerName="switch-graph-a625d"
Sep 17 21:01:08.240635 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240489    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kube-rbac-proxy"
Sep 17 21:01:08.240635 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.240495    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2bd31b-92e4-47d4-8109-ecec6131c122" containerName="kserve-container"
Sep 17 21:01:08.243362 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.243343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.249940 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.249923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 17 21:01:08.250719 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.250701    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 17 21:01:08.260108 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.260087    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"]
Sep 17 21:01:08.366267 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.366235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dae8ee01-f900-41f6-b909-c722612ca93f-openshift-service-ca-bundle\") pod \"model-chainer-58db8ff9c8-zlxhc\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") " pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.366425 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.366288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dae8ee01-f900-41f6-b909-c722612ca93f-proxy-tls\") pod \"model-chainer-58db8ff9c8-zlxhc\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") " pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.467362 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.467311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dae8ee01-f900-41f6-b909-c722612ca93f-openshift-service-ca-bundle\") pod \"model-chainer-58db8ff9c8-zlxhc\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") " pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.467518 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.467403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dae8ee01-f900-41f6-b909-c722612ca93f-proxy-tls\") pod \"model-chainer-58db8ff9c8-zlxhc\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") " pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.467986 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.467968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dae8ee01-f900-41f6-b909-c722612ca93f-openshift-service-ca-bundle\") pod \"model-chainer-58db8ff9c8-zlxhc\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") " pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.469845 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.469830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dae8ee01-f900-41f6-b909-c722612ca93f-proxy-tls\") pod \"model-chainer-58db8ff9c8-zlxhc\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") " pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.552618 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.552542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:08.822042 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:08.822018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"]
Sep 17 21:01:08.824613 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:01:08.824586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae8ee01_f900_41f6_b909_c722612ca93f.slice/crio-9144d0b6e0e9295df17052e1c8dea11ddb26aad5f6c44ac979cd52c3c0076138 WatchSource:0}: Error finding container 9144d0b6e0e9295df17052e1c8dea11ddb26aad5f6c44ac979cd52c3c0076138: Status 404 returned error can't find the container with id 9144d0b6e0e9295df17052e1c8dea11ddb26aad5f6c44ac979cd52c3c0076138
Sep 17 21:01:09.399644 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:09.399610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" event={"ID":"dae8ee01-f900-41f6-b909-c722612ca93f","Type":"ContainerStarted","Data":"54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364"}
Sep 17 21:01:09.400026 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:09.399665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" event={"ID":"dae8ee01-f900-41f6-b909-c722612ca93f","Type":"ContainerStarted","Data":"9144d0b6e0e9295df17052e1c8dea11ddb26aad5f6c44ac979cd52c3c0076138"}
Sep 17 21:01:09.400026 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:09.399698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:09.417131 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:09.417087    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podStartSLOduration=1.417073147 podStartE2EDuration="1.417073147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:09.41586835 +0000 UTC m=+709.109677698" watchObservedRunningTime="2026-09-17 21:01:09.417073147 +0000 UTC m=+709.110882477"
Sep 17 21:01:10.203764 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:10.203730    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:01:15.409026 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:15.408995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:19.757231 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:19.757197    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"]
Sep 17 21:01:19.757725 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:19.757443    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer" containerID="cri-o://54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364" gracePeriod=30
Sep 17 21:01:20.003155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.003115    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"]
Sep 17 21:01:20.006836 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.006811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.009676 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.009621    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ed647-kube-rbac-proxy-sar-config\""
Sep 17 21:01:20.009676 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.009622    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ed647-predictor-serving-cert\""
Sep 17 21:01:20.017155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.017134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"]
Sep 17 21:01:20.165142 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.165104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.165142 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.165153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.165420 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.165201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6rb\" (UniqueName: \"kubernetes.io/projected/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-api-access-wj6rb\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.204303 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.204272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:01:20.266093 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.266012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.266093 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.266050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.266093 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.266089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6rb\" (UniqueName: \"kubernetes.io/projected/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-api-access-wj6rb\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.266367 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:01:20.266180    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ed647-predictor-serving-cert: secret "success-200-isvc-ed647-predictor-serving-cert" not found
Sep 17 21:01:20.266367 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:01:20.266267    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls podName:4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6 nodeName:}" failed. No retries permitted until 2026-09-17 21:01:20.766247593 +0000 UTC m=+720.460056902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls") pod "success-200-isvc-ed647-predictor-7fdb57868c-fgq77" (UID: "4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6") : secret "success-200-isvc-ed647-predictor-serving-cert" not found
Sep 17 21:01:20.266608 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.266589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.278431 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.278403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6rb\" (UniqueName: \"kubernetes.io/projected/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-api-access-wj6rb\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.407651 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.407614    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:20.769980 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.769938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.772522 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.772501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls\") pod \"success-200-isvc-ed647-predictor-7fdb57868c-fgq77\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") " pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:20.918010 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:20.917969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:21.186881 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:21.186850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"]
Sep 17 21:01:21.189778 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:01:21.189751    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc81c65_e92e_4ae8_aa9a_f70e151a1fc6.slice/crio-7d328fa1dd7da4b7559b4e209e45c5e1631dd5d87f7c00fcbfd196fac9710729 WatchSource:0}: Error finding container 7d328fa1dd7da4b7559b4e209e45c5e1631dd5d87f7c00fcbfd196fac9710729: Status 404 returned error can't find the container with id 7d328fa1dd7da4b7559b4e209e45c5e1631dd5d87f7c00fcbfd196fac9710729
Sep 17 21:01:21.455243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:21.455150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" event={"ID":"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6","Type":"ContainerStarted","Data":"a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53"}
Sep 17 21:01:21.455243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:21.455204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" event={"ID":"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6","Type":"ContainerStarted","Data":"25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd"}
Sep 17 21:01:21.455243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:21.455225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" event={"ID":"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6","Type":"ContainerStarted","Data":"7d328fa1dd7da4b7559b4e209e45c5e1631dd5d87f7c00fcbfd196fac9710729"}
Sep 17 21:01:21.455488 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:21.455288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:21.473641 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:21.473586    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podStartSLOduration=2.473571042 podStartE2EDuration="2.473571042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:21.472412419 +0000 UTC m=+721.166221769" watchObservedRunningTime="2026-09-17 21:01:21.473571042 +0000 UTC m=+721.167380373"
Sep 17 21:01:22.462610 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:22.462577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:22.463969 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:22.463946    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:01:23.466754 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:23.466709    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:01:25.407893 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:25.407851    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:28.471738 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:28.471711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:01:28.472178 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:28.472149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:01:30.407448 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:30.407411    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:30.407818 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:30.407512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:33.399934 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.399900    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"]
Sep 17 21:01:33.404408 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.404387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.406768 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.406747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-23363-serving-cert\""
Sep 17 21:01:33.406901 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.406885    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-23363-kube-rbac-proxy-sar-config\""
Sep 17 21:01:33.413154 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.413131    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"]
Sep 17 21:01:33.469870 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.469836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2cc4bc-0b37-4d94-a117-985a114cf23f-proxy-tls\") pod \"switch-graph-23363-67f7d4f7ff-mp2w6\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") " pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.470050 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.469885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2cc4bc-0b37-4d94-a117-985a114cf23f-openshift-service-ca-bundle\") pod \"switch-graph-23363-67f7d4f7ff-mp2w6\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") " pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.570501 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.570471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2cc4bc-0b37-4d94-a117-985a114cf23f-proxy-tls\") pod \"switch-graph-23363-67f7d4f7ff-mp2w6\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") " pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.570709 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.570515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2cc4bc-0b37-4d94-a117-985a114cf23f-openshift-service-ca-bundle\") pod \"switch-graph-23363-67f7d4f7ff-mp2w6\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") " pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.571128 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.571099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2cc4bc-0b37-4d94-a117-985a114cf23f-openshift-service-ca-bundle\") pod \"switch-graph-23363-67f7d4f7ff-mp2w6\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") " pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.573051 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.573028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2cc4bc-0b37-4d94-a117-985a114cf23f-proxy-tls\") pod \"switch-graph-23363-67f7d4f7ff-mp2w6\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") " pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.715904 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.715825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:33.987450 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:33.987422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"]
Sep 17 21:01:33.990612 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:01:33.990586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2cc4bc_0b37_4d94_a117_985a114cf23f.slice/crio-a1348e83fc0bf183d96d7e06e85a85015e07b36c91db7ad14ee7d3405544a805 WatchSource:0}: Error finding container a1348e83fc0bf183d96d7e06e85a85015e07b36c91db7ad14ee7d3405544a805: Status 404 returned error can't find the container with id a1348e83fc0bf183d96d7e06e85a85015e07b36c91db7ad14ee7d3405544a805
Sep 17 21:01:34.515795 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:34.515763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" event={"ID":"2f2cc4bc-0b37-4d94-a117-985a114cf23f","Type":"ContainerStarted","Data":"ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5"}
Sep 17 21:01:34.516160 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:34.515810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" event={"ID":"2f2cc4bc-0b37-4d94-a117-985a114cf23f","Type":"ContainerStarted","Data":"a1348e83fc0bf183d96d7e06e85a85015e07b36c91db7ad14ee7d3405544a805"}
Sep 17 21:01:34.516160 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:34.515842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:34.533657 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:34.533614    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podStartSLOduration=1.533592743 podStartE2EDuration="1.533592743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:34.532556884 +0000 UTC m=+734.226366214" watchObservedRunningTime="2026-09-17 21:01:34.533592743 +0000 UTC m=+734.227402088"
Sep 17 21:01:35.407316 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:35.407281    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:38.473119 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:38.473079    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:01:40.407913 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:40.407874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:40.526359 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:40.526332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:01:45.408009 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:45.407970    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:48.472708 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:48.472671    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:01:49.930384 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:49.930359    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:50.006255 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.006225    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dae8ee01-f900-41f6-b909-c722612ca93f-proxy-tls\") pod \"dae8ee01-f900-41f6-b909-c722612ca93f\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") "
Sep 17 21:01:50.006430 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.006263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dae8ee01-f900-41f6-b909-c722612ca93f-openshift-service-ca-bundle\") pod \"dae8ee01-f900-41f6-b909-c722612ca93f\" (UID: \"dae8ee01-f900-41f6-b909-c722612ca93f\") "
Sep 17 21:01:50.006635 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.006613    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae8ee01-f900-41f6-b909-c722612ca93f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "dae8ee01-f900-41f6-b909-c722612ca93f" (UID: "dae8ee01-f900-41f6-b909-c722612ca93f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:01:50.008442 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.008417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae8ee01-f900-41f6-b909-c722612ca93f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dae8ee01-f900-41f6-b909-c722612ca93f" (UID: "dae8ee01-f900-41f6-b909-c722612ca93f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:01:50.107642 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.107612    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dae8ee01-f900-41f6-b909-c722612ca93f-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:01:50.107642 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.107639    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dae8ee01-f900-41f6-b909-c722612ca93f-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:01:50.599340 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.599296    2579 generic.go:356] "Generic (PLEG): container finished" podID="dae8ee01-f900-41f6-b909-c722612ca93f" containerID="54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364" exitCode=137
Sep 17 21:01:50.599503 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.599353    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"
Sep 17 21:01:50.599503 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.599386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" event={"ID":"dae8ee01-f900-41f6-b909-c722612ca93f","Type":"ContainerDied","Data":"54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364"}
Sep 17 21:01:50.599503 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.599442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc" event={"ID":"dae8ee01-f900-41f6-b909-c722612ca93f","Type":"ContainerDied","Data":"9144d0b6e0e9295df17052e1c8dea11ddb26aad5f6c44ac979cd52c3c0076138"}
Sep 17 21:01:50.599503 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.599465    2579 scope.go:117] "RemoveContainer" containerID="54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364"
Sep 17 21:01:50.607999 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.607976    2579 scope.go:117] "RemoveContainer" containerID="54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364"
Sep 17 21:01:50.608280 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:01:50.608261    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364\": container with ID starting with 54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364 not found: ID does not exist" containerID="54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364"
Sep 17 21:01:50.608357 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.608291    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364"} err="failed to get container status \"54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364\": rpc error: code = NotFound desc = could not find container \"54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364\": container with ID starting with 54202d131cc47ea2f1a866c8ff083537d57e022d5f008b2423b3a6fd5b725364 not found: ID does not exist"
Sep 17 21:01:50.614701 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.614680    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"]
Sep 17 21:01:50.618090 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:50.618067    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc"]
Sep 17 21:01:52.300019 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:52.299979    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" path="/var/lib/kubelet/pods/dae8ee01-f900-41f6-b909-c722612ca93f/volumes"
Sep 17 21:01:58.472307 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:01:58.472269    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:02:08.473084 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:08.473048    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:02:18.473138 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:18.473108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:02:29.979399 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.979366    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"]
Sep 17 21:02:29.979776 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.979719    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer"
Sep 17 21:02:29.979776 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.979730    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer"
Sep 17 21:02:29.979849 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.979796    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae8ee01-f900-41f6-b909-c722612ca93f" containerName="model-chainer"
Sep 17 21:02:29.984046 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.984018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:29.986786 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.986769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ed647-kube-rbac-proxy-sar-config\""
Sep 17 21:02:29.987007 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.986980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ed647-serving-cert\""
Sep 17 21:02:29.993391 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:29.993363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"]
Sep 17 21:02:30.044132 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.044085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655fcdc1-36fd-4c23-8e9c-ab35565e228f-proxy-tls\") pod \"sequence-graph-ed647-5445897977-p9svl\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") " pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.044373 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.044174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655fcdc1-36fd-4c23-8e9c-ab35565e228f-openshift-service-ca-bundle\") pod \"sequence-graph-ed647-5445897977-p9svl\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") " pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.145372 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.145340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655fcdc1-36fd-4c23-8e9c-ab35565e228f-proxy-tls\") pod \"sequence-graph-ed647-5445897977-p9svl\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") " pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.145563 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.145394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655fcdc1-36fd-4c23-8e9c-ab35565e228f-openshift-service-ca-bundle\") pod \"sequence-graph-ed647-5445897977-p9svl\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") " pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.146042 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.146016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655fcdc1-36fd-4c23-8e9c-ab35565e228f-openshift-service-ca-bundle\") pod \"sequence-graph-ed647-5445897977-p9svl\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") " pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.147939 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.147923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655fcdc1-36fd-4c23-8e9c-ab35565e228f-proxy-tls\") pod \"sequence-graph-ed647-5445897977-p9svl\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") " pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.296851 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.296741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.567896 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.567872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"]
Sep 17 21:02:30.569896 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:02:30.569868    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655fcdc1_36fd_4c23_8e9c_ab35565e228f.slice/crio-1027bb2802dc3ed314e34465d481299b2d79b2565de9eb33e44705bfceb36e77 WatchSource:0}: Error finding container 1027bb2802dc3ed314e34465d481299b2d79b2565de9eb33e44705bfceb36e77: Status 404 returned error can't find the container with id 1027bb2802dc3ed314e34465d481299b2d79b2565de9eb33e44705bfceb36e77
Sep 17 21:02:30.780960 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.780923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" event={"ID":"655fcdc1-36fd-4c23-8e9c-ab35565e228f","Type":"ContainerStarted","Data":"071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133"}
Sep 17 21:02:30.781135 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.780981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" event={"ID":"655fcdc1-36fd-4c23-8e9c-ab35565e228f","Type":"ContainerStarted","Data":"1027bb2802dc3ed314e34465d481299b2d79b2565de9eb33e44705bfceb36e77"}
Sep 17 21:02:30.781135 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.781060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:02:30.796906 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:30.796853    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podStartSLOduration=1.796836681 podStartE2EDuration="1.796836681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:02:30.796018767 +0000 UTC m=+790.489828098" watchObservedRunningTime="2026-09-17 21:02:30.796836681 +0000 UTC m=+790.490646013"
Sep 17 21:02:36.790952 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:02:36.790873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:04:22.273699 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:04:22.273673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:04:22.274688 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:04:22.274668    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:04:22.278445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:04:22.278413    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:04:22.279497 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:04:22.279477    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:09:22.305149 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:22.305122    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:09:22.306400 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:22.306377    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:09:22.310226 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:22.310207    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:09:22.311464 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:22.311447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:09:48.450915 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.450881    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"]
Sep 17 21:09:48.451392 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.451108    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363" containerID="cri-o://ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5" gracePeriod=30
Sep 17 21:09:48.589799 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.589765    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"]
Sep 17 21:09:48.590093 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.590065    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" containerID="cri-o://ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db" gracePeriod=30
Sep 17 21:09:48.590194 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.590107    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kube-rbac-proxy" containerID="cri-o://3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff" gracePeriod=30
Sep 17 21:09:48.667924 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.667891    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"]
Sep 17 21:09:48.671743 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.671722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.674279 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.674234    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-46851-kube-rbac-proxy-sar-config\""
Sep 17 21:09:48.674279 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.674241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-46851-predictor-serving-cert\""
Sep 17 21:09:48.682580 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.682557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"]
Sep 17 21:09:48.758974 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.758939    2579 generic.go:356] "Generic (PLEG): container finished" podID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerID="3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff" exitCode=2
Sep 17 21:09:48.759146 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.759007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" event={"ID":"25e17d71-83fc-41f5-aee2-a16cf61edb83","Type":"ContainerDied","Data":"3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff"}
Sep 17 21:09:48.767486 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.767453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-proxy-tls\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.767624 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.767504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.767624 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.767535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrjx\" (UniqueName: \"kubernetes.io/projected/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-api-access-whrjx\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.868646 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.868611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.868646 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.868648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrjx\" (UniqueName: \"kubernetes.io/projected/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-api-access-whrjx\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.868876 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.868739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-proxy-tls\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.869279 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.869254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.871311 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.871281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-proxy-tls\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.877266 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.877247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrjx\" (UniqueName: \"kubernetes.io/projected/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-api-access-whrjx\") pod \"success-200-isvc-46851-predictor-5c8c955cdc-c8zks\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") " pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:48.985088 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:48.985007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:49.259969 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.259946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"]
Sep 17 21:09:49.262656 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:09:49.262632    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef4969d_38c2_4bd1_aeeb_9eff2f89d7d8.slice/crio-b382604bbbaf2b16208872d26a7c408b6a7fed67310406c2443b4a3d7c1e8829 WatchSource:0}: Error finding container b382604bbbaf2b16208872d26a7c408b6a7fed67310406c2443b4a3d7c1e8829: Status 404 returned error can't find the container with id b382604bbbaf2b16208872d26a7c408b6a7fed67310406c2443b4a3d7c1e8829
Sep 17 21:09:49.765933 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.765898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" event={"ID":"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8","Type":"ContainerStarted","Data":"7b89e02cd95b5ab8c3e508c58a3751964068061de738faca46d4f68a51ccab1b"}
Sep 17 21:09:49.766388 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.765961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" event={"ID":"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8","Type":"ContainerStarted","Data":"9f2f5d4bee1635ea2617ea5b7bc33b795859d7c76363c1ed7e7e9aabf4d884a8"}
Sep 17 21:09:49.766388 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.765989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" event={"ID":"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8","Type":"ContainerStarted","Data":"b382604bbbaf2b16208872d26a7c408b6a7fed67310406c2443b4a3d7c1e8829"}
Sep 17 21:09:49.766388 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.766013    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:49.766388 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.766036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:49.767611 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.767589    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 21:09:49.784479 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:49.784433    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podStartSLOduration=1.784415348 podStartE2EDuration="1.784415348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:09:49.782089192 +0000 UTC m=+1229.475898551" watchObservedRunningTime="2026-09-17 21:09:49.784415348 +0000 UTC m=+1229.478224680"
Sep 17 21:09:50.199027 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:50.198989    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 21:09:50.203255 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:50.203234    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 21:09:50.525080 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:50.524991    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:09:50.770552 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:50.770510    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 21:09:52.054111 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.054086    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:09:52.096616 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.096579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvjst\" (UniqueName: \"kubernetes.io/projected/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-api-access-jvjst\") pod \"25e17d71-83fc-41f5-aee2-a16cf61edb83\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") "
Sep 17 21:09:52.096805 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.096642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-rbac-proxy-sar-config\") pod \"25e17d71-83fc-41f5-aee2-a16cf61edb83\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") "
Sep 17 21:09:52.096805 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.096714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e17d71-83fc-41f5-aee2-a16cf61edb83-proxy-tls\") pod \"25e17d71-83fc-41f5-aee2-a16cf61edb83\" (UID: \"25e17d71-83fc-41f5-aee2-a16cf61edb83\") "
Sep 17 21:09:52.097004 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.096979    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25e17d71-83fc-41f5-aee2-a16cf61edb83" (UID: "25e17d71-83fc-41f5-aee2-a16cf61edb83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:52.098859 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.098833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e17d71-83fc-41f5-aee2-a16cf61edb83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25e17d71-83fc-41f5-aee2-a16cf61edb83" (UID: "25e17d71-83fc-41f5-aee2-a16cf61edb83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:09:52.099036 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.099018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-api-access-jvjst" (OuterVolumeSpecName: "kube-api-access-jvjst") pod "25e17d71-83fc-41f5-aee2-a16cf61edb83" (UID: "25e17d71-83fc-41f5-aee2-a16cf61edb83"). InnerVolumeSpecName "kube-api-access-jvjst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:09:52.198174 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.198095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjst\" (UniqueName: \"kubernetes.io/projected/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-api-access-jvjst\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:09:52.198174 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.198122    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e17d71-83fc-41f5-aee2-a16cf61edb83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:09:52.198174 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.198133    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e17d71-83fc-41f5-aee2-a16cf61edb83-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:09:52.788371 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.788310    2579 generic.go:356] "Generic (PLEG): container finished" podID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerID="ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db" exitCode=0
Sep 17 21:09:52.788567 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.788393    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"
Sep 17 21:09:52.788567 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.788394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" event={"ID":"25e17d71-83fc-41f5-aee2-a16cf61edb83","Type":"ContainerDied","Data":"ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db"}
Sep 17 21:09:52.788567 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.788497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn" event={"ID":"25e17d71-83fc-41f5-aee2-a16cf61edb83","Type":"ContainerDied","Data":"170a67a7979b3d906733b2170f2221c34ecbe66835569728220547449becdc98"}
Sep 17 21:09:52.788567 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.788533    2579 scope.go:117] "RemoveContainer" containerID="3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff"
Sep 17 21:09:52.797062 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.797045    2579 scope.go:117] "RemoveContainer" containerID="ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db"
Sep 17 21:09:52.804897 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.804879    2579 scope.go:117] "RemoveContainer" containerID="3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff"
Sep 17 21:09:52.805146 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:09:52.805125    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff\": container with ID starting with 3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff not found: ID does not exist" containerID="3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff"
Sep 17 21:09:52.805230 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.805155    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff"} err="failed to get container status \"3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff\": rpc error: code = NotFound desc = could not find container \"3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff\": container with ID starting with 3325c4595b104eddc208caa060ecd20db559dd4bc0436900f3ae3678895498ff not found: ID does not exist"
Sep 17 21:09:52.805230 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.805174    2579 scope.go:117] "RemoveContainer" containerID="ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db"
Sep 17 21:09:52.805509 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:09:52.805493    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db\": container with ID starting with ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db not found: ID does not exist" containerID="ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db"
Sep 17 21:09:52.805608 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.805514    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db"} err="failed to get container status \"ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db\": rpc error: code = NotFound desc = could not find container \"ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db\": container with ID starting with ea2f4a5ce4faa105198462bfcbbe28abcf45c03df072cb88d7607c690b6727db not found: ID does not exist"
Sep 17 21:09:52.806054 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.806038    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"]
Sep 17 21:09:52.809812 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:52.809794    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn"]
Sep 17 21:09:54.299564 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:54.299530    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" path="/var/lib/kubelet/pods/25e17d71-83fc-41f5-aee2-a16cf61edb83/volumes"
Sep 17 21:09:55.525083 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:55.525045    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:09:55.775835 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:55.775751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:09:55.776277 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:09:55.776248    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 21:10:00.525141 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:00.525097    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:10:00.527536 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:00.525207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:10:05.525762 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:05.525681    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:10:05.776425 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:05.776339    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 21:10:10.525004 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:10.524966    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:10:15.525199 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:15.525155    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:10:15.776909 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:15.776828    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 21:10:18.639445 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.639414    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:10:18.709828 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.709752    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2cc4bc-0b37-4d94-a117-985a114cf23f-proxy-tls\") pod \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") "
Sep 17 21:10:18.709828 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.709816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2cc4bc-0b37-4d94-a117-985a114cf23f-openshift-service-ca-bundle\") pod \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\" (UID: \"2f2cc4bc-0b37-4d94-a117-985a114cf23f\") "
Sep 17 21:10:18.710107 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.710083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2cc4bc-0b37-4d94-a117-985a114cf23f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2f2cc4bc-0b37-4d94-a117-985a114cf23f" (UID: "2f2cc4bc-0b37-4d94-a117-985a114cf23f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:10:18.711924 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.711903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2cc4bc-0b37-4d94-a117-985a114cf23f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f2cc4bc-0b37-4d94-a117-985a114cf23f" (UID: "2f2cc4bc-0b37-4d94-a117-985a114cf23f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:10:18.811117 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.811086    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2cc4bc-0b37-4d94-a117-985a114cf23f-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:10:18.811117 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.811112    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2cc4bc-0b37-4d94-a117-985a114cf23f-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:10:18.915633 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.915601    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerID="ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5" exitCode=0
Sep 17 21:10:18.915780 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.915655    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"
Sep 17 21:10:18.915780 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.915657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" event={"ID":"2f2cc4bc-0b37-4d94-a117-985a114cf23f","Type":"ContainerDied","Data":"ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5"}
Sep 17 21:10:18.915855 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.915796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6" event={"ID":"2f2cc4bc-0b37-4d94-a117-985a114cf23f","Type":"ContainerDied","Data":"a1348e83fc0bf183d96d7e06e85a85015e07b36c91db7ad14ee7d3405544a805"}
Sep 17 21:10:18.915855 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.915818    2579 scope.go:117] "RemoveContainer" containerID="ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5"
Sep 17 21:10:18.924808 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.924780    2579 scope.go:117] "RemoveContainer" containerID="ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5"
Sep 17 21:10:18.925078 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:10:18.925059    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5\": container with ID starting with ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5 not found: ID does not exist" containerID="ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5"
Sep 17 21:10:18.925140 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.925087    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5"} err="failed to get container status \"ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5\": rpc error: code = NotFound desc = could not find container \"ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5\": container with ID starting with ea398535152a38016be0476ab045aedb23e6b170b7d52e9921aad9edc62e4fd5 not found: ID does not exist"
Sep 17 21:10:18.937136 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.937111    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"]
Sep 17 21:10:18.943191 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:18.943171    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6"]
Sep 17 21:10:20.299089 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:20.299053    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" path="/var/lib/kubelet/pods/2f2cc4bc-0b37-4d94-a117-985a114cf23f/volumes"
Sep 17 21:10:25.776451 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:25.776408    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 21:10:35.776302 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:35.776260    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 21:10:45.532944 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.532913    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"]
Sep 17 21:10:45.533428 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.533171    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647" containerID="cri-o://071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133" gracePeriod=30
Sep 17 21:10:45.629091 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.629055    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"]
Sep 17 21:10:45.629456 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.629403    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" containerID="cri-o://25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd" gracePeriod=30
Sep 17 21:10:45.629559 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.629449    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kube-rbac-proxy" containerID="cri-o://a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53" gracePeriod=30
Sep 17 21:10:45.762521 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.762484    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"]
Sep 17 21:10:45.762903 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.762885    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363"
Sep 17 21:10:45.762986 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.762905    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363"
Sep 17 21:10:45.762986 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.762919    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kube-rbac-proxy"
Sep 17 21:10:45.762986 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.762926    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kube-rbac-proxy"
Sep 17 21:10:45.762986 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.762955    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container"
Sep 17 21:10:45.762986 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.762963    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container"
Sep 17 21:10:45.763234 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.763038    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kserve-container"
Sep 17 21:10:45.763234 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.763051    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e17d71-83fc-41f5-aee2-a16cf61edb83" containerName="kube-rbac-proxy"
Sep 17 21:10:45.763234 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.763064    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2cc4bc-0b37-4d94-a117-985a114cf23f" containerName="switch-graph-23363"
Sep 17 21:10:45.766272 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.766250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.768579 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.768557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-62e6f-kube-rbac-proxy-sar-config\""
Sep 17 21:10:45.768688 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.768623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-62e6f-predictor-serving-cert\""
Sep 17 21:10:45.774779 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.774753    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"]
Sep 17 21:10:45.776783 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.776764    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:10:45.837644 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.837611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdjt8\" (UniqueName: \"kubernetes.io/projected/62ed77d4-3462-4062-b61e-044e392f64c1-kube-api-access-wdjt8\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.837844 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.837664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.837844 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.837732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ed77d4-3462-4062-b61e-044e392f64c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.938303 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.938264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdjt8\" (UniqueName: \"kubernetes.io/projected/62ed77d4-3462-4062-b61e-044e392f64c1-kube-api-access-wdjt8\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.938303 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.938305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.938559 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.938362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ed77d4-3462-4062-b61e-044e392f64c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.938559 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:10:45.938492    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-serving-cert: secret "success-200-isvc-62e6f-predictor-serving-cert" not found
Sep 17 21:10:45.938559 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:10:45.938557    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls podName:62ed77d4-3462-4062-b61e-044e392f64c1 nodeName:}" failed. No retries permitted until 2026-09-17 21:10:46.438538515 +0000 UTC m=+1286.132347840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls") pod "success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" (UID: "62ed77d4-3462-4062-b61e-044e392f64c1") : secret "success-200-isvc-62e6f-predictor-serving-cert" not found
Sep 17 21:10:45.939037 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.939016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ed77d4-3462-4062-b61e-044e392f64c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:45.952348 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:45.952295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdjt8\" (UniqueName: \"kubernetes.io/projected/62ed77d4-3462-4062-b61e-044e392f64c1-kube-api-access-wdjt8\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:46.038942 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:46.038910    2579 generic.go:356] "Generic (PLEG): container finished" podID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerID="a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53" exitCode=2
Sep 17 21:10:46.039084 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:46.038958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" event={"ID":"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6","Type":"ContainerDied","Data":"a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53"}
Sep 17 21:10:46.441476 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:46.441440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:46.444036 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:46.444003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls\") pod \"success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") " pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:46.678804 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:46.678771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:46.790040 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:46.789992    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:10:46.950500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:46.950474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"]
Sep 17 21:10:46.952603 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:10:46.952574    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ed77d4_3462_4062_b61e_044e392f64c1.slice/crio-25ca74fad643f645c0bcde25988df5799fbf7ae1dafe23c1ef0feaea39e7445a WatchSource:0}: Error finding container 25ca74fad643f645c0bcde25988df5799fbf7ae1dafe23c1ef0feaea39e7445a: Status 404 returned error can't find the container with id 25ca74fad643f645c0bcde25988df5799fbf7ae1dafe23c1ef0feaea39e7445a
Sep 17 21:10:47.045850 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:47.045824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" event={"ID":"62ed77d4-3462-4062-b61e-044e392f64c1","Type":"ContainerStarted","Data":"929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265"}
Sep 17 21:10:47.045942 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:47.045878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" event={"ID":"62ed77d4-3462-4062-b61e-044e392f64c1","Type":"ContainerStarted","Data":"25ca74fad643f645c0bcde25988df5799fbf7ae1dafe23c1ef0feaea39e7445a"}
Sep 17 21:10:48.051919 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:48.051886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" event={"ID":"62ed77d4-3462-4062-b61e-044e392f64c1","Type":"ContainerStarted","Data":"6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641"}
Sep 17 21:10:48.052315 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:48.052008    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:48.070056 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:48.070011    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podStartSLOduration=3.069996629 podStartE2EDuration="3.069996629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:10:48.067590845 +0000 UTC m=+1287.761400173" watchObservedRunningTime="2026-09-17 21:10:48.069996629 +0000 UTC m=+1287.763805959"
Sep 17 21:10:48.467177 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:48.467139    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 21:10:48.472499 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:48.472475    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 21:10:48.979558 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:48.979535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:10:49.058212 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.058122    2579 generic.go:356] "Generic (PLEG): container finished" podID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerID="25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd" exitCode=0
Sep 17 21:10:49.058212 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.058174    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"
Sep 17 21:10:49.058212 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.058200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" event={"ID":"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6","Type":"ContainerDied","Data":"25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd"}
Sep 17 21:10:49.058674 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.058258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77" event={"ID":"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6","Type":"ContainerDied","Data":"7d328fa1dd7da4b7559b4e209e45c5e1631dd5d87f7c00fcbfd196fac9710729"}
Sep 17 21:10:49.058674 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.058287    2579 scope.go:117] "RemoveContainer" containerID="a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53"
Sep 17 21:10:49.058674 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.058573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:49.060204 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.060180    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 21:10:49.061478 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.061456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-rbac-proxy-sar-config\") pod \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") "
Sep 17 21:10:49.061588 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.061491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls\") pod \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") "
Sep 17 21:10:49.061588 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.061527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj6rb\" (UniqueName: \"kubernetes.io/projected/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-api-access-wj6rb\") pod \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\" (UID: \"4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6\") "
Sep 17 21:10:49.061834 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.061808    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" (UID: "4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:10:49.063700 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.063677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" (UID: "4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:10:49.064135 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.064112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-api-access-wj6rb" (OuterVolumeSpecName: "kube-api-access-wj6rb") pod "4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" (UID: "4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6"). InnerVolumeSpecName "kube-api-access-wj6rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:10:49.067142 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.067113    2579 scope.go:117] "RemoveContainer" containerID="25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd"
Sep 17 21:10:49.078449 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.078426    2579 scope.go:117] "RemoveContainer" containerID="a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53"
Sep 17 21:10:49.078707 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:10:49.078687    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53\": container with ID starting with a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53 not found: ID does not exist" containerID="a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53"
Sep 17 21:10:49.078745 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.078719    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53"} err="failed to get container status \"a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53\": rpc error: code = NotFound desc = could not find container \"a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53\": container with ID starting with a1048a55161a74404c60a10f991079e6f9bf7f8979aa3788f628731c30647c53 not found: ID does not exist"
Sep 17 21:10:49.078745 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.078737    2579 scope.go:117] "RemoveContainer" containerID="25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd"
Sep 17 21:10:49.078991 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:10:49.078973    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd\": container with ID starting with 25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd not found: ID does not exist" containerID="25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd"
Sep 17 21:10:49.079035 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.078997    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd"} err="failed to get container status \"25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd\": rpc error: code = NotFound desc = could not find container \"25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd\": container with ID starting with 25ab323f79b8f204635ef1ad2ab4f2f62ede4ce49b080e8711077bccd07c81fd not found: ID does not exist"
Sep 17 21:10:49.162755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.162728    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:10:49.162755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.162749    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:10:49.162755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.162759    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj6rb\" (UniqueName: \"kubernetes.io/projected/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6-kube-api-access-wj6rb\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:10:49.382670 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.382634    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"]
Sep 17 21:10:49.384293 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:49.384272    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77"]
Sep 17 21:10:50.070231 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:50.070194    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 21:10:50.299738 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:50.299706    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" path="/var/lib/kubelet/pods/4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6/volumes"
Sep 17 21:10:51.790606 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:51.790569    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:10:55.076365 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:55.076314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:10:55.076754 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:55.076728    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 21:10:56.790608 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:56.790561    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:10:56.791012 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:56.790706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:10:58.781267 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.781190    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"]
Sep 17 21:10:58.781681 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.781555    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container"
Sep 17 21:10:58.781681 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.781568    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container"
Sep 17 21:10:58.781681 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.781592    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kube-rbac-proxy"
Sep 17 21:10:58.781681 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.781598    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kube-rbac-proxy"
Sep 17 21:10:58.781681 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.781676    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kube-rbac-proxy"
Sep 17 21:10:58.781835 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.781687    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc81c65-e92e-4ae8-aa9a-f70e151a1fc6" containerName="kserve-container"
Sep 17 21:10:58.786183 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.786165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:58.789340 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.789305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-46851-serving-cert\""
Sep 17 21:10:58.789481 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.789368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-46851-kube-rbac-proxy-sar-config\""
Sep 17 21:10:58.817155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.817125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"]
Sep 17 21:10:58.849515 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.849484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29b5fa1-8a00-4bf9-981e-caac3c9de967-openshift-service-ca-bundle\") pod \"ensemble-graph-46851-555c4ddb89-n7mwm\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") " pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:58.849680 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.849533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f29b5fa1-8a00-4bf9-981e-caac3c9de967-proxy-tls\") pod \"ensemble-graph-46851-555c4ddb89-n7mwm\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") " pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:58.950425 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.950393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29b5fa1-8a00-4bf9-981e-caac3c9de967-openshift-service-ca-bundle\") pod \"ensemble-graph-46851-555c4ddb89-n7mwm\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") " pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:58.950606 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.950448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f29b5fa1-8a00-4bf9-981e-caac3c9de967-proxy-tls\") pod \"ensemble-graph-46851-555c4ddb89-n7mwm\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") " pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:58.951057 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.951035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29b5fa1-8a00-4bf9-981e-caac3c9de967-openshift-service-ca-bundle\") pod \"ensemble-graph-46851-555c4ddb89-n7mwm\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") " pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:58.952991 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:58.952971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f29b5fa1-8a00-4bf9-981e-caac3c9de967-proxy-tls\") pod \"ensemble-graph-46851-555c4ddb89-n7mwm\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") " pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:59.096522 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:59.096493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:10:59.365554 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:10:59.365526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"]
Sep 17 21:10:59.368021 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:10:59.367995    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29b5fa1_8a00_4bf9_981e_caac3c9de967.slice/crio-a4a7e30d93514b6eb23f72022bbdb28df44aea31702234c9702705d87a301b37 WatchSource:0}: Error finding container a4a7e30d93514b6eb23f72022bbdb28df44aea31702234c9702705d87a301b37: Status 404 returned error can't find the container with id a4a7e30d93514b6eb23f72022bbdb28df44aea31702234c9702705d87a301b37
Sep 17 21:11:00.114487 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:00.114443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" event={"ID":"f29b5fa1-8a00-4bf9-981e-caac3c9de967","Type":"ContainerStarted","Data":"279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277"}
Sep 17 21:11:00.114956 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:00.114503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" event={"ID":"f29b5fa1-8a00-4bf9-981e-caac3c9de967","Type":"ContainerStarted","Data":"a4a7e30d93514b6eb23f72022bbdb28df44aea31702234c9702705d87a301b37"}
Sep 17 21:11:00.114956 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:00.114584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:11:00.130199 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:00.130149    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podStartSLOduration=2.130135469 podStartE2EDuration="2.130135469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:11:00.129814913 +0000 UTC m=+1299.823624243" watchObservedRunningTime="2026-09-17 21:11:00.130135469 +0000 UTC m=+1299.823944799"
Sep 17 21:11:01.789619 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:01.789584    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:05.077054 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:05.077017    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 21:11:06.124182 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:06.124150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:11:06.789405 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:06.789360    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:09.375604 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.375572    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"]
Sep 17 21:11:09.376109 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.375753    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851" containerID="cri-o://279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277" gracePeriod=30
Sep 17 21:11:09.478523 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.478494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"]
Sep 17 21:11:09.478914 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.478727    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container" containerID="cri-o://9f2f5d4bee1635ea2617ea5b7bc33b795859d7c76363c1ed7e7e9aabf4d884a8" gracePeriod=30
Sep 17 21:11:09.478914 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.478770    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kube-rbac-proxy" containerID="cri-o://7b89e02cd95b5ab8c3e508c58a3751964068061de738faca46d4f68a51ccab1b" gracePeriod=30
Sep 17 21:11:09.609436 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.609400    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"]
Sep 17 21:11:09.612771 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.612753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.615167 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.615140    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31e6a-kube-rbac-proxy-sar-config\""
Sep 17 21:11:09.615275 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.615136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31e6a-predictor-serving-cert\""
Sep 17 21:11:09.626253 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.626204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"]
Sep 17 21:11:09.744164 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.744118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjmwc\" (UniqueName: \"kubernetes.io/projected/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-api-access-gjmwc\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.744343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.744265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.744343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.744309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.845144 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.845115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.845144 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.845147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.845396 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.845203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjmwc\" (UniqueName: \"kubernetes.io/projected/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-api-access-gjmwc\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.845396 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:11:09.845294    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-serving-cert: secret "success-200-isvc-31e6a-predictor-serving-cert" not found
Sep 17 21:11:09.845509 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:11:09.845402    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls podName:a9f46079-fee3-4582-9c6e-9b2e0403b0fa nodeName:}" failed. No retries permitted until 2026-09-17 21:11:10.345373816 +0000 UTC m=+1310.039183131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls") pod "success-200-isvc-31e6a-predictor-6c64485579-ztxgg" (UID: "a9f46079-fee3-4582-9c6e-9b2e0403b0fa") : secret "success-200-isvc-31e6a-predictor-serving-cert" not found
Sep 17 21:11:09.845906 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.845882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:09.857940 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:09.857907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjmwc\" (UniqueName: \"kubernetes.io/projected/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-api-access-gjmwc\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:10.160596 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:10.160561    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerID="7b89e02cd95b5ab8c3e508c58a3751964068061de738faca46d4f68a51ccab1b" exitCode=2
Sep 17 21:11:10.160761 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:10.160626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" event={"ID":"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8","Type":"ContainerDied","Data":"7b89e02cd95b5ab8c3e508c58a3751964068061de738faca46d4f68a51ccab1b"}
Sep 17 21:11:10.350027 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:10.349992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:10.352723 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:10.352696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls\") pod \"success-200-isvc-31e6a-predictor-6c64485579-ztxgg\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") " pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:10.522726 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:10.522628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:10.771084 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:10.771049    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 21:11:10.796050 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:10.796021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"]
Sep 17 21:11:10.797940 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:11:10.797904    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f46079_fee3_4582_9c6e_9b2e0403b0fa.slice/crio-8a5f5f491977a230d7b53bc609b080e3e0b2affd8ea0a2b65417ebf537e8dbd1 WatchSource:0}: Error finding container 8a5f5f491977a230d7b53bc609b080e3e0b2affd8ea0a2b65417ebf537e8dbd1: Status 404 returned error can't find the container with id 8a5f5f491977a230d7b53bc609b080e3e0b2affd8ea0a2b65417ebf537e8dbd1
Sep 17 21:11:11.123198 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:11.123158    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:11.167102 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:11.167059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" event={"ID":"a9f46079-fee3-4582-9c6e-9b2e0403b0fa","Type":"ContainerStarted","Data":"afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063"}
Sep 17 21:11:11.167260 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:11.167161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" event={"ID":"a9f46079-fee3-4582-9c6e-9b2e0403b0fa","Type":"ContainerStarted","Data":"b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1"}
Sep 17 21:11:11.167260 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:11.167194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:11.167260 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:11.167223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" event={"ID":"a9f46079-fee3-4582-9c6e-9b2e0403b0fa","Type":"ContainerStarted","Data":"8a5f5f491977a230d7b53bc609b080e3e0b2affd8ea0a2b65417ebf537e8dbd1"}
Sep 17 21:11:11.184260 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:11.184214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podStartSLOduration=2.18419965 podStartE2EDuration="2.18419965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:11:11.183280072 +0000 UTC m=+1310.877089404" watchObservedRunningTime="2026-09-17 21:11:11.18419965 +0000 UTC m=+1310.878009036"
Sep 17 21:11:11.790815 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:11.790762    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:12.172771 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:12.172744    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:12.173849 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:12.173820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:11:13.178962 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.178931    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerID="9f2f5d4bee1635ea2617ea5b7bc33b795859d7c76363c1ed7e7e9aabf4d884a8" exitCode=0
Sep 17 21:11:13.179339 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.179004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" event={"ID":"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8","Type":"ContainerDied","Data":"9f2f5d4bee1635ea2617ea5b7bc33b795859d7c76363c1ed7e7e9aabf4d884a8"}
Sep 17 21:11:13.179453 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.179431    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:11:13.227017 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.226991    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:11:13.376158 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.376128    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whrjx\" (UniqueName: \"kubernetes.io/projected/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-api-access-whrjx\") pod \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") "
Sep 17 21:11:13.376346 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.376182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-rbac-proxy-sar-config\") pod \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") "
Sep 17 21:11:13.376346 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.376263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-proxy-tls\") pod \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\" (UID: \"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8\") "
Sep 17 21:11:13.376627 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.376596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" (UID: "5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:11:13.378509 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.378487    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" (UID: "5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:11:13.378602 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.378491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-api-access-whrjx" (OuterVolumeSpecName: "kube-api-access-whrjx") pod "5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" (UID: "5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8"). InnerVolumeSpecName "kube-api-access-whrjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:11:13.477314 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.477284    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:11:13.477314 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.477312    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrjx\" (UniqueName: \"kubernetes.io/projected/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-api-access-whrjx\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:11:13.477519 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:13.477346    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:11:14.184744 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:14.184718    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"
Sep 17 21:11:14.185136 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:14.184718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks" event={"ID":"5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8","Type":"ContainerDied","Data":"b382604bbbaf2b16208872d26a7c408b6a7fed67310406c2443b4a3d7c1e8829"}
Sep 17 21:11:14.185136 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:14.184870    2579 scope.go:117] "RemoveContainer" containerID="7b89e02cd95b5ab8c3e508c58a3751964068061de738faca46d4f68a51ccab1b"
Sep 17 21:11:14.195073 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:14.195044    2579 scope.go:117] "RemoveContainer" containerID="9f2f5d4bee1635ea2617ea5b7bc33b795859d7c76363c1ed7e7e9aabf4d884a8"
Sep 17 21:11:14.207110 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:14.207085    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"]
Sep 17 21:11:14.211225 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:14.211203    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks"]
Sep 17 21:11:14.299509 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:14.299479    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" path="/var/lib/kubelet/pods/5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8/volumes"
Sep 17 21:11:15.077474 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:15.077432    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 21:11:16.125022 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.124987    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:16.181719 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.181695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:11:16.200148 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.200121    2579 generic.go:356] "Generic (PLEG): container finished" podID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerID="071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133" exitCode=0
Sep 17 21:11:16.200286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.200161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" event={"ID":"655fcdc1-36fd-4c23-8e9c-ab35565e228f","Type":"ContainerDied","Data":"071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133"}
Sep 17 21:11:16.200286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.200166    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"
Sep 17 21:11:16.200286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.200214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl" event={"ID":"655fcdc1-36fd-4c23-8e9c-ab35565e228f","Type":"ContainerDied","Data":"1027bb2802dc3ed314e34465d481299b2d79b2565de9eb33e44705bfceb36e77"}
Sep 17 21:11:16.200286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.200244    2579 scope.go:117] "RemoveContainer" containerID="071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133"
Sep 17 21:11:16.208597 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.208576    2579 scope.go:117] "RemoveContainer" containerID="071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133"
Sep 17 21:11:16.208873 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:11:16.208855    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133\": container with ID starting with 071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133 not found: ID does not exist" containerID="071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133"
Sep 17 21:11:16.208935 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.208883    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133"} err="failed to get container status \"071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133\": rpc error: code = NotFound desc = could not find container \"071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133\": container with ID starting with 071b112181d52a149f2dbf182d8d5abf2f8fdf2aa97621f3338bfdd46c912133 not found: ID does not exist"
Sep 17 21:11:16.301833 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.301760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655fcdc1-36fd-4c23-8e9c-ab35565e228f-openshift-service-ca-bundle\") pod \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") "
Sep 17 21:11:16.301833 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.301800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655fcdc1-36fd-4c23-8e9c-ab35565e228f-proxy-tls\") pod \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\" (UID: \"655fcdc1-36fd-4c23-8e9c-ab35565e228f\") "
Sep 17 21:11:16.302126 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.302104    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655fcdc1-36fd-4c23-8e9c-ab35565e228f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "655fcdc1-36fd-4c23-8e9c-ab35565e228f" (UID: "655fcdc1-36fd-4c23-8e9c-ab35565e228f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:11:16.304019 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.303991    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655fcdc1-36fd-4c23-8e9c-ab35565e228f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "655fcdc1-36fd-4c23-8e9c-ab35565e228f" (UID: "655fcdc1-36fd-4c23-8e9c-ab35565e228f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:11:16.403391 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.403352    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/655fcdc1-36fd-4c23-8e9c-ab35565e228f-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:11:16.403391 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.403382    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655fcdc1-36fd-4c23-8e9c-ab35565e228f-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:11:16.522182 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.522154    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"]
Sep 17 21:11:16.525237 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:16.525208    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl"]
Sep 17 21:11:18.184361 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:18.184306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:11:18.184807 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:18.184765    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:11:18.300823 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:18.300789    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" path="/var/lib/kubelet/pods/655fcdc1-36fd-4c23-8e9c-ab35565e228f/volumes"
Sep 17 21:11:21.123565 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:21.123527    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:21.124031 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:21.123657    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:11:25.076802 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:25.076764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 21:11:26.122823 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:26.122778    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:28.185290 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:28.185254    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:11:31.123411 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:31.123368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:35.077644 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:35.077560    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 21:11:36.122837 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:36.122799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:11:38.185630 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:38.185588    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:11:39.522414 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:39.522390    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:11:39.584538 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:39.584507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f29b5fa1-8a00-4bf9-981e-caac3c9de967-proxy-tls\") pod \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") "
Sep 17 21:11:39.584700 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:39.584549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29b5fa1-8a00-4bf9-981e-caac3c9de967-openshift-service-ca-bundle\") pod \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\" (UID: \"f29b5fa1-8a00-4bf9-981e-caac3c9de967\") "
Sep 17 21:11:39.584952 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:39.584922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29b5fa1-8a00-4bf9-981e-caac3c9de967-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f29b5fa1-8a00-4bf9-981e-caac3c9de967" (UID: "f29b5fa1-8a00-4bf9-981e-caac3c9de967"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:11:39.586810 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:39.586785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29b5fa1-8a00-4bf9-981e-caac3c9de967-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f29b5fa1-8a00-4bf9-981e-caac3c9de967" (UID: "f29b5fa1-8a00-4bf9-981e-caac3c9de967"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:11:39.685406 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:39.685305    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f29b5fa1-8a00-4bf9-981e-caac3c9de967-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:11:39.685406 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:39.685352    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29b5fa1-8a00-4bf9-981e-caac3c9de967-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:11:40.307889 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.307855    2579 generic.go:356] "Generic (PLEG): container finished" podID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerID="279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277" exitCode=0
Sep 17 21:11:40.308049 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.307899    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"
Sep 17 21:11:40.308049 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.307924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" event={"ID":"f29b5fa1-8a00-4bf9-981e-caac3c9de967","Type":"ContainerDied","Data":"279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277"}
Sep 17 21:11:40.308049 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.307989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm" event={"ID":"f29b5fa1-8a00-4bf9-981e-caac3c9de967","Type":"ContainerDied","Data":"a4a7e30d93514b6eb23f72022bbdb28df44aea31702234c9702705d87a301b37"}
Sep 17 21:11:40.308049 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.308017    2579 scope.go:117] "RemoveContainer" containerID="279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277"
Sep 17 21:11:40.316463 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.316445    2579 scope.go:117] "RemoveContainer" containerID="279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277"
Sep 17 21:11:40.316736 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:11:40.316716    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277\": container with ID starting with 279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277 not found: ID does not exist" containerID="279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277"
Sep 17 21:11:40.316793 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.316745    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277"} err="failed to get container status \"279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277\": rpc error: code = NotFound desc = could not find container \"279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277\": container with ID starting with 279f1a53fc0a116a21eb2fde6c3d206cf7f5d975d32fa17ef7d0ef895dfd9277 not found: ID does not exist"
Sep 17 21:11:40.323090 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.323067    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"]
Sep 17 21:11:40.328154 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:40.328133    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm"]
Sep 17 21:11:42.300252 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:42.300219    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" path="/var/lib/kubelet/pods/f29b5fa1-8a00-4bf9-981e-caac3c9de967/volumes"
Sep 17 21:11:45.078211 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:45.078174    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:11:48.185737 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:48.185700    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:11:55.767640 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.767600    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"]
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.767955    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.767967    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.767981    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.767986    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.767994    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kube-rbac-proxy"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.768000    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kube-rbac-proxy"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.768009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.768014    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.768063    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kserve-container"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.768070    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef4969d-38c2-4bd1-aeeb-9eff2f89d7d8" containerName="kube-rbac-proxy"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.768078    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29b5fa1-8a00-4bf9-981e-caac3c9de967" containerName="ensemble-graph-46851"
Sep 17 21:11:55.768155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.768085    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="655fcdc1-36fd-4c23-8e9c-ab35565e228f" containerName="sequence-graph-ed647"
Sep 17 21:11:55.772366 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.772346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:55.774744 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.774723    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-62e6f-kube-rbac-proxy-sar-config\""
Sep 17 21:11:55.774826 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.774760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-62e6f-serving-cert\""
Sep 17 21:11:55.783669 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.783647    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"]
Sep 17 21:11:55.816791 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.816751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-openshift-service-ca-bundle\") pod \"sequence-graph-62e6f-7c8567c89d-p74m9\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") " pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:55.816957 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.816908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls\") pod \"sequence-graph-62e6f-7c8567c89d-p74m9\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") " pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:55.918339 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.918278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-openshift-service-ca-bundle\") pod \"sequence-graph-62e6f-7c8567c89d-p74m9\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") " pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:55.918530 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.918390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls\") pod \"sequence-graph-62e6f-7c8567c89d-p74m9\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") " pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:55.918530 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:11:55.918523    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-62e6f-serving-cert: secret "sequence-graph-62e6f-serving-cert" not found
Sep 17 21:11:55.918655 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:11:55.918602    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls podName:07d3d8b2-c1eb-4916-997d-b1c3d4c22353 nodeName:}" failed. No retries permitted until 2026-09-17 21:11:56.418581268 +0000 UTC m=+1356.112390583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls") pod "sequence-graph-62e6f-7c8567c89d-p74m9" (UID: "07d3d8b2-c1eb-4916-997d-b1c3d4c22353") : secret "sequence-graph-62e6f-serving-cert" not found
Sep 17 21:11:55.919030 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:55.919004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-openshift-service-ca-bundle\") pod \"sequence-graph-62e6f-7c8567c89d-p74m9\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") " pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:56.423265 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:56.423232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls\") pod \"sequence-graph-62e6f-7c8567c89d-p74m9\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") " pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:56.425762 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:56.425745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls\") pod \"sequence-graph-62e6f-7c8567c89d-p74m9\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") " pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:56.683525 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:56.683438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:56.957274 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:56.957248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"]
Sep 17 21:11:56.959377 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:11:56.959346    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d3d8b2_c1eb_4916_997d_b1c3d4c22353.slice/crio-75af587598d06fbf2b50147f30eccbd8d81d344f4ae9f4fc8aece17eedd6cbb9 WatchSource:0}: Error finding container 75af587598d06fbf2b50147f30eccbd8d81d344f4ae9f4fc8aece17eedd6cbb9: Status 404 returned error can't find the container with id 75af587598d06fbf2b50147f30eccbd8d81d344f4ae9f4fc8aece17eedd6cbb9
Sep 17 21:11:57.381858 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:57.381826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" event={"ID":"07d3d8b2-c1eb-4916-997d-b1c3d4c22353","Type":"ContainerStarted","Data":"64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5"}
Sep 17 21:11:57.382026 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:57.381875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" event={"ID":"07d3d8b2-c1eb-4916-997d-b1c3d4c22353","Type":"ContainerStarted","Data":"75af587598d06fbf2b50147f30eccbd8d81d344f4ae9f4fc8aece17eedd6cbb9"}
Sep 17 21:11:57.382026 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:57.381906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:11:57.398008 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:57.397945    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podStartSLOduration=2.397924844 podStartE2EDuration="2.397924844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:11:57.39723032 +0000 UTC m=+1357.091039653" watchObservedRunningTime="2026-09-17 21:11:57.397924844 +0000 UTC m=+1357.091734177"
Sep 17 21:11:58.185438 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:11:58.185404    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:12:03.391201 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:03.391164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:12:06.799805 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:06.799772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"]
Sep 17 21:12:06.800194 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:06.799950    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f" containerID="cri-o://64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5" gracePeriod=30
Sep 17 21:12:07.150192 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.150155    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"]
Sep 17 21:12:07.150479 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.150453    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container" containerID="cri-o://929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265" gracePeriod=30
Sep 17 21:12:07.150571 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.150540    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kube-rbac-proxy" containerID="cri-o://6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641" gracePeriod=30
Sep 17 21:12:07.209446 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.209414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"]
Sep 17 21:12:07.213170 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.213149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.215568 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.215542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b5a48-predictor-serving-cert\""
Sep 17 21:12:07.215732 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.215709    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b5a48-kube-rbac-proxy-sar-config\""
Sep 17 21:12:07.228808 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.225339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"]
Sep 17 21:12:07.319834 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.319801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczdr\" (UniqueName: \"kubernetes.io/projected/509f7877-02a4-4acf-85ab-672c783f5118-kube-api-access-wczdr\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.320014 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.319868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/509f7877-02a4-4acf-85ab-672c783f5118-proxy-tls\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.320014 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.319903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/509f7877-02a4-4acf-85ab-672c783f5118-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.421095 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.421006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczdr\" (UniqueName: \"kubernetes.io/projected/509f7877-02a4-4acf-85ab-672c783f5118-kube-api-access-wczdr\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.421095 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.421079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/509f7877-02a4-4acf-85ab-672c783f5118-proxy-tls\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.421386 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.421110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/509f7877-02a4-4acf-85ab-672c783f5118-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.421998 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.421972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/509f7877-02a4-4acf-85ab-672c783f5118-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.424226 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.424199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/509f7877-02a4-4acf-85ab-672c783f5118-proxy-tls\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.429009 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.428981    2579 generic.go:356] "Generic (PLEG): container finished" podID="62ed77d4-3462-4062-b61e-044e392f64c1" containerID="6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641" exitCode=2
Sep 17 21:12:07.429138 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.429027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" event={"ID":"62ed77d4-3462-4062-b61e-044e392f64c1","Type":"ContainerDied","Data":"6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641"}
Sep 17 21:12:07.429778 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.429759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczdr\" (UniqueName: \"kubernetes.io/projected/509f7877-02a4-4acf-85ab-672c783f5118-kube-api-access-wczdr\") pod \"success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") " pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.528423 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.528392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:07.797813 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:07.797785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"]
Sep 17 21:12:07.800791 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:12:07.800757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509f7877_02a4_4acf_85ab_672c783f5118.slice/crio-c692c4f496f1041c5ffe7a664db29f33b5913e47bddbb43dda74a1b52777915d WatchSource:0}: Error finding container c692c4f496f1041c5ffe7a664db29f33b5913e47bddbb43dda74a1b52777915d: Status 404 returned error can't find the container with id c692c4f496f1041c5ffe7a664db29f33b5913e47bddbb43dda74a1b52777915d
Sep 17 21:12:08.185960 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.185936    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:12:08.390565 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.390521    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:12:08.435364 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.435309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" event={"ID":"509f7877-02a4-4acf-85ab-672c783f5118","Type":"ContainerStarted","Data":"2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859"}
Sep 17 21:12:08.435535 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.435393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" event={"ID":"509f7877-02a4-4acf-85ab-672c783f5118","Type":"ContainerStarted","Data":"ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e"}
Sep 17 21:12:08.435535 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.435428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" event={"ID":"509f7877-02a4-4acf-85ab-672c783f5118","Type":"ContainerStarted","Data":"c692c4f496f1041c5ffe7a664db29f33b5913e47bddbb43dda74a1b52777915d"}
Sep 17 21:12:08.435535 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.435463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:08.435535 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.435492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:08.436923 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.436867    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:12:08.462639 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:08.462596    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podStartSLOduration=1.46258307 podStartE2EDuration="1.46258307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:12:08.462170722 +0000 UTC m=+1368.155980054" watchObservedRunningTime="2026-09-17 21:12:08.46258307 +0000 UTC m=+1368.156392400"
Sep 17 21:12:09.440233 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:09.440195    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:12:10.070744 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.070708    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 21:12:10.788931 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.788902    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:12:10.849150 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.849086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ed77d4-3462-4062-b61e-044e392f64c1-kube-rbac-proxy-sar-config\") pod \"62ed77d4-3462-4062-b61e-044e392f64c1\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") "
Sep 17 21:12:10.849283 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.849158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls\") pod \"62ed77d4-3462-4062-b61e-044e392f64c1\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") "
Sep 17 21:12:10.849283 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.849178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdjt8\" (UniqueName: \"kubernetes.io/projected/62ed77d4-3462-4062-b61e-044e392f64c1-kube-api-access-wdjt8\") pod \"62ed77d4-3462-4062-b61e-044e392f64c1\" (UID: \"62ed77d4-3462-4062-b61e-044e392f64c1\") "
Sep 17 21:12:10.849446 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.849422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ed77d4-3462-4062-b61e-044e392f64c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62ed77d4-3462-4062-b61e-044e392f64c1" (UID: "62ed77d4-3462-4062-b61e-044e392f64c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:12:10.851355 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.851314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62ed77d4-3462-4062-b61e-044e392f64c1" (UID: "62ed77d4-3462-4062-b61e-044e392f64c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:12:10.851355 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.851341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ed77d4-3462-4062-b61e-044e392f64c1-kube-api-access-wdjt8" (OuterVolumeSpecName: "kube-api-access-wdjt8") pod "62ed77d4-3462-4062-b61e-044e392f64c1" (UID: "62ed77d4-3462-4062-b61e-044e392f64c1"). InnerVolumeSpecName "kube-api-access-wdjt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:12:10.949845 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.949816    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ed77d4-3462-4062-b61e-044e392f64c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:12:10.949845 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.949840    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ed77d4-3462-4062-b61e-044e392f64c1-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:12:10.949845 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:10.949851    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdjt8\" (UniqueName: \"kubernetes.io/projected/62ed77d4-3462-4062-b61e-044e392f64c1-kube-api-access-wdjt8\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:12:11.450092 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.450054    2579 generic.go:356] "Generic (PLEG): container finished" podID="62ed77d4-3462-4062-b61e-044e392f64c1" containerID="929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265" exitCode=0
Sep 17 21:12:11.450286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.450105    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"
Sep 17 21:12:11.450286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.450135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" event={"ID":"62ed77d4-3462-4062-b61e-044e392f64c1","Type":"ContainerDied","Data":"929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265"}
Sep 17 21:12:11.450286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.450193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r" event={"ID":"62ed77d4-3462-4062-b61e-044e392f64c1","Type":"ContainerDied","Data":"25ca74fad643f645c0bcde25988df5799fbf7ae1dafe23c1ef0feaea39e7445a"}
Sep 17 21:12:11.450286 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.450218    2579 scope.go:117] "RemoveContainer" containerID="6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641"
Sep 17 21:12:11.459202 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.459187    2579 scope.go:117] "RemoveContainer" containerID="929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265"
Sep 17 21:12:11.466441 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.466424    2579 scope.go:117] "RemoveContainer" containerID="6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641"
Sep 17 21:12:11.466689 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:12:11.466670    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641\": container with ID starting with 6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641 not found: ID does not exist" containerID="6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641"
Sep 17 21:12:11.466767 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.466700    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641"} err="failed to get container status \"6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641\": rpc error: code = NotFound desc = could not find container \"6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641\": container with ID starting with 6eab21179e37b923121b9e5fb001ecc34f679e06d37b73dc70d1876430704641 not found: ID does not exist"
Sep 17 21:12:11.466767 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.466723    2579 scope.go:117] "RemoveContainer" containerID="929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265"
Sep 17 21:12:11.467003 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:12:11.466986    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265\": container with ID starting with 929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265 not found: ID does not exist" containerID="929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265"
Sep 17 21:12:11.467043 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.467009    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265"} err="failed to get container status \"929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265\": rpc error: code = NotFound desc = could not find container \"929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265\": container with ID starting with 929c51e73b9d61fa42ef788af78d22c5a47cd5180f2902a7638e27c8df303265 not found: ID does not exist"
Sep 17 21:12:11.471169 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.471147    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"]
Sep 17 21:12:11.477600 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:11.477578    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r"]
Sep 17 21:12:12.299074 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:12.299045    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" path="/var/lib/kubelet/pods/62ed77d4-3462-4062-b61e-044e392f64c1/volumes"
Sep 17 21:12:13.389641 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:13.389603    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:12:14.445345 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:14.445303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:12:14.445901 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:14.445882    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:12:18.391350 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:18.391297    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:12:18.391805 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:18.391432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:12:19.657441 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.657408    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"]
Sep 17 21:12:19.657806 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.657762    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container"
Sep 17 21:12:19.657806 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.657773    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container"
Sep 17 21:12:19.657806 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.657789    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kube-rbac-proxy"
Sep 17 21:12:19.657806 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.657795    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kube-rbac-proxy"
Sep 17 21:12:19.657942 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.657844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kserve-container"
Sep 17 21:12:19.657942 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.657854    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ed77d4-3462-4062-b61e-044e392f64c1" containerName="kube-rbac-proxy"
Sep 17 21:12:19.669013 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.668987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:19.672626 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.672597    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"]
Sep 17 21:12:19.672888 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.672860    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-31e6a-serving-cert\""
Sep 17 21:12:19.673143 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.673117    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-31e6a-kube-rbac-proxy-sar-config\""
Sep 17 21:12:19.826799 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.826767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5933b7-d062-49ac-9557-f07dc1aa19b9-openshift-service-ca-bundle\") pod \"ensemble-graph-31e6a-b7ddd645d-m4f2b\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") " pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:19.826978 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.826875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls\") pod \"ensemble-graph-31e6a-b7ddd645d-m4f2b\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") " pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:19.927372 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.927268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls\") pod \"ensemble-graph-31e6a-b7ddd645d-m4f2b\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") " pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:19.927372 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.927358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5933b7-d062-49ac-9557-f07dc1aa19b9-openshift-service-ca-bundle\") pod \"ensemble-graph-31e6a-b7ddd645d-m4f2b\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") " pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:19.927577 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:12:19.927430    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-31e6a-serving-cert: secret "ensemble-graph-31e6a-serving-cert" not found
Sep 17 21:12:19.927577 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:12:19.927500    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls podName:4a5933b7-d062-49ac-9557-f07dc1aa19b9 nodeName:}" failed. No retries permitted until 2026-09-17 21:12:20.427479109 +0000 UTC m=+1380.121288422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls") pod "ensemble-graph-31e6a-b7ddd645d-m4f2b" (UID: "4a5933b7-d062-49ac-9557-f07dc1aa19b9") : secret "ensemble-graph-31e6a-serving-cert" not found
Sep 17 21:12:19.927925 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:19.927897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5933b7-d062-49ac-9557-f07dc1aa19b9-openshift-service-ca-bundle\") pod \"ensemble-graph-31e6a-b7ddd645d-m4f2b\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") " pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:20.431308 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:20.431278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls\") pod \"ensemble-graph-31e6a-b7ddd645d-m4f2b\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") " pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:20.433888 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:20.433864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls\") pod \"ensemble-graph-31e6a-b7ddd645d-m4f2b\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") " pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:20.581678 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:20.581641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:20.852808 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:20.852780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"]
Sep 17 21:12:20.854835 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:12:20.854807    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5933b7_d062_49ac_9557_f07dc1aa19b9.slice/crio-a415e5ece1ddd3f90924bb8373168da092ab6153e8886ae15e1035591ae9a156 WatchSource:0}: Error finding container a415e5ece1ddd3f90924bb8373168da092ab6153e8886ae15e1035591ae9a156: Status 404 returned error can't find the container with id a415e5ece1ddd3f90924bb8373168da092ab6153e8886ae15e1035591ae9a156
Sep 17 21:12:21.496934 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:21.496904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" event={"ID":"4a5933b7-d062-49ac-9557-f07dc1aa19b9","Type":"ContainerStarted","Data":"3c9ce95817f49e50c6e67822e549a4b90cf372cde3b4bb2bc3d370aef135b292"}
Sep 17 21:12:21.497109 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:21.496955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" event={"ID":"4a5933b7-d062-49ac-9557-f07dc1aa19b9","Type":"ContainerStarted","Data":"a415e5ece1ddd3f90924bb8373168da092ab6153e8886ae15e1035591ae9a156"}
Sep 17 21:12:21.497109 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:21.497046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:21.514860 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:21.514810    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podStartSLOduration=2.514795822 podStartE2EDuration="2.514795822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:12:21.512410248 +0000 UTC m=+1381.206219591" watchObservedRunningTime="2026-09-17 21:12:21.514795822 +0000 UTC m=+1381.208605180"
Sep 17 21:12:23.389542 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:23.389506    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:12:24.446713 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:24.446676    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:12:27.507819 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:27.507791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:12:28.390046 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:28.390009    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:12:33.390535 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:33.390495    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:12:34.446642 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:34.446597    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:12:37.448419 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.448394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:12:37.475921 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.475894    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-openshift-service-ca-bundle\") pod \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") "
Sep 17 21:12:37.476073 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.476017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls\") pod \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\" (UID: \"07d3d8b2-c1eb-4916-997d-b1c3d4c22353\") "
Sep 17 21:12:37.476271 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.476243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "07d3d8b2-c1eb-4916-997d-b1c3d4c22353" (UID: "07d3d8b2-c1eb-4916-997d-b1c3d4c22353"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:12:37.478153 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.478127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07d3d8b2-c1eb-4916-997d-b1c3d4c22353" (UID: "07d3d8b2-c1eb-4916-997d-b1c3d4c22353"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:12:37.571446 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.571361    2579 generic.go:356] "Generic (PLEG): container finished" podID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerID="64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5" exitCode=0
Sep 17 21:12:37.571446 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.571402    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"
Sep 17 21:12:37.571649 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.571445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" event={"ID":"07d3d8b2-c1eb-4916-997d-b1c3d4c22353","Type":"ContainerDied","Data":"64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5"}
Sep 17 21:12:37.571649 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.571519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9" event={"ID":"07d3d8b2-c1eb-4916-997d-b1c3d4c22353","Type":"ContainerDied","Data":"75af587598d06fbf2b50147f30eccbd8d81d344f4ae9f4fc8aece17eedd6cbb9"}
Sep 17 21:12:37.571649 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.571542    2579 scope.go:117] "RemoveContainer" containerID="64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5"
Sep 17 21:12:37.576701 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.576680    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:12:37.576701 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.576701    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d3d8b2-c1eb-4916-997d-b1c3d4c22353-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:12:37.580618 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.580601    2579 scope.go:117] "RemoveContainer" containerID="64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5"
Sep 17 21:12:37.580885 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:12:37.580868    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5\": container with ID starting with 64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5 not found: ID does not exist" containerID="64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5"
Sep 17 21:12:37.580934 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.580895    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5"} err="failed to get container status \"64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5\": rpc error: code = NotFound desc = could not find container \"64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5\": container with ID starting with 64ac974e14ced9cc11d69fbfd3f44612683cf22c3bbf40bfaa77166caf0716a5 not found: ID does not exist"
Sep 17 21:12:37.592797 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.592769    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"]
Sep 17 21:12:37.594546 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:37.594529    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9"]
Sep 17 21:12:38.299199 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:38.299164    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" path="/var/lib/kubelet/pods/07d3d8b2-c1eb-4916-997d-b1c3d4c22353/volumes"
Sep 17 21:12:44.445886 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:44.445852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:12:54.445972 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:12:54.445932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:13:04.447571 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:04.447489    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:13:17.217758 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.217718    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"]
Sep 17 21:13:17.218207 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.218062    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f"
Sep 17 21:13:17.218207 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.218072    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f"
Sep 17 21:13:17.218207 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.218136    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d3d8b2-c1eb-4916-997d-b1c3d4c22353" containerName="sequence-graph-62e6f"
Sep 17 21:13:17.220992 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.220975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.223879 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.223858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b5a48-serving-cert\""
Sep 17 21:13:17.224040 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.224023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b5a48-kube-rbac-proxy-sar-config\""
Sep 17 21:13:17.233868 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.233844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"]
Sep 17 21:13:17.314623 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.314594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-openshift-service-ca-bundle\") pod \"sequence-graph-b5a48-848c5fd7d8-nzfp4\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") " pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.314791 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.314642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-proxy-tls\") pod \"sequence-graph-b5a48-848c5fd7d8-nzfp4\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") " pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.415768 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.415725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-openshift-service-ca-bundle\") pod \"sequence-graph-b5a48-848c5fd7d8-nzfp4\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") " pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.415975 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.415787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-proxy-tls\") pod \"sequence-graph-b5a48-848c5fd7d8-nzfp4\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") " pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.416396 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.416359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-openshift-service-ca-bundle\") pod \"sequence-graph-b5a48-848c5fd7d8-nzfp4\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") " pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.418381 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.418358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-proxy-tls\") pod \"sequence-graph-b5a48-848c5fd7d8-nzfp4\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") " pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.531603 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.531517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:17.818278 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:17.818190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"]
Sep 17 21:13:17.821751 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:13:17.821721    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d086a6_ed2b_4a98_a64d_a49808f7e6cd.slice/crio-ade226ede50b23ac0497cb52a6398e9a309d2815279cd54afa17e7dd8d231e15 WatchSource:0}: Error finding container ade226ede50b23ac0497cb52a6398e9a309d2815279cd54afa17e7dd8d231e15: Status 404 returned error can't find the container with id ade226ede50b23ac0497cb52a6398e9a309d2815279cd54afa17e7dd8d231e15
Sep 17 21:13:18.759300 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:18.759250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" event={"ID":"27d086a6-ed2b-4a98-a64d-a49808f7e6cd","Type":"ContainerStarted","Data":"b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428"}
Sep 17 21:13:18.759743 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:18.759359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" event={"ID":"27d086a6-ed2b-4a98-a64d-a49808f7e6cd","Type":"ContainerStarted","Data":"ade226ede50b23ac0497cb52a6398e9a309d2815279cd54afa17e7dd8d231e15"}
Sep 17 21:13:18.759743 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:18.759500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:13:18.775857 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:18.775817    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podStartSLOduration=1.7758046360000002 podStartE2EDuration="1.775804636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:13:18.774638726 +0000 UTC m=+1438.468448057" watchObservedRunningTime="2026-09-17 21:13:18.775804636 +0000 UTC m=+1438.469613967"
Sep 17 21:13:24.770029 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:13:24.769995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:14:22.334727 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:14:22.334700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:14:22.337734 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:14:22.337715    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:14:22.339122 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:14:22.339100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:14:22.342186 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:14:22.342169    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:19:22.363853 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:19:22.363825    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:19:22.366199 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:19:22.366177    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:19:22.368569 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:19:22.368548    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:19:22.371086 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:19:22.371070    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:20:34.841514 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:34.841413    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"]
Sep 17 21:20:34.844155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:34.841656    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a" containerID="cri-o://3c9ce95817f49e50c6e67822e549a4b90cf372cde3b4bb2bc3d370aef135b292" gracePeriod=30
Sep 17 21:20:34.998580 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:34.998548    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"]
Sep 17 21:20:34.998830 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:34.998807    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" containerID="cri-o://b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1" gracePeriod=30
Sep 17 21:20:34.998934 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:34.998904    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kube-rbac-proxy" containerID="cri-o://afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063" gracePeriod=30
Sep 17 21:20:35.175930 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.175900    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"]
Sep 17 21:20:35.179368 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.179345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.185584 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.185558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d3600-kube-rbac-proxy-sar-config\""
Sep 17 21:20:35.187652 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.187632    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d3600-predictor-serving-cert\""
Sep 17 21:20:35.197675 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.196450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"]
Sep 17 21:20:35.305834 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.305800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.306034 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.305880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcdb\" (UniqueName: \"kubernetes.io/projected/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-api-access-4mcdb\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.306034 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.305963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.406536 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.406442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcdb\" (UniqueName: \"kubernetes.io/projected/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-api-access-4mcdb\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.406536 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.406507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.406746 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.406549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.406746 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:20:35.406658    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d3600-predictor-serving-cert: secret "success-200-isvc-d3600-predictor-serving-cert" not found
Sep 17 21:20:35.406746 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:20:35.406730    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls podName:1941a697-469b-4130-8b62-e7fbd3ef0eed nodeName:}" failed. No retries permitted until 2026-09-17 21:20:35.906709647 +0000 UTC m=+1875.600518961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls") pod "success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" (UID: "1941a697-469b-4130-8b62-e7fbd3ef0eed") : secret "success-200-isvc-d3600-predictor-serving-cert" not found
Sep 17 21:20:35.407174 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.407153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.416068 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.416044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcdb\" (UniqueName: \"kubernetes.io/projected/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-api-access-4mcdb\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.727469 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.727379    2579 generic.go:356] "Generic (PLEG): container finished" podID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerID="afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063" exitCode=2
Sep 17 21:20:35.727612 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.727457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" event={"ID":"a9f46079-fee3-4582-9c6e-9b2e0403b0fa","Type":"ContainerDied","Data":"afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063"}
Sep 17 21:20:35.910288 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:35.910256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:35.910671 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:20:35.910414    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d3600-predictor-serving-cert: secret "success-200-isvc-d3600-predictor-serving-cert" not found
Sep 17 21:20:35.910671 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:20:35.910487    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls podName:1941a697-469b-4130-8b62-e7fbd3ef0eed nodeName:}" failed. No retries permitted until 2026-09-17 21:20:36.910471732 +0000 UTC m=+1876.604281055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls") pod "success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" (UID: "1941a697-469b-4130-8b62-e7fbd3ef0eed") : secret "success-200-isvc-d3600-predictor-serving-cert" not found
Sep 17 21:20:36.919643 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:36.919613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:36.922145 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:36.922123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls\") pod \"success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") " pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:36.997703 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:36.997673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:37.279760 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:37.279736    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"]
Sep 17 21:20:37.282151 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:20:37.282120    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1941a697_469b_4130_8b62_e7fbd3ef0eed.slice/crio-3851cd788ba3630bfddda0681faf4eff38a010102b49c88d1659431d1fac630f WatchSource:0}: Error finding container 3851cd788ba3630bfddda0681faf4eff38a010102b49c88d1659431d1fac630f: Status 404 returned error can't find the container with id 3851cd788ba3630bfddda0681faf4eff38a010102b49c88d1659431d1fac630f
Sep 17 21:20:37.505679 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:37.505631    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:20:37.737610 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:37.737575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" event={"ID":"1941a697-469b-4130-8b62-e7fbd3ef0eed","Type":"ContainerStarted","Data":"0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560"}
Sep 17 21:20:37.737801 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:37.737635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" event={"ID":"1941a697-469b-4130-8b62-e7fbd3ef0eed","Type":"ContainerStarted","Data":"09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712"}
Sep 17 21:20:37.737801 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:37.737670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:37.737801 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:37.737689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" event={"ID":"1941a697-469b-4130-8b62-e7fbd3ef0eed","Type":"ContainerStarted","Data":"3851cd788ba3630bfddda0681faf4eff38a010102b49c88d1659431d1fac630f"}
Sep 17 21:20:38.179694 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.179652    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 21:20:38.184988 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.184965    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 21:20:38.348413 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.348391    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:20:38.404267 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.404219    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podStartSLOduration=3.404203572 podStartE2EDuration="3.404203572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:20:37.764581919 +0000 UTC m=+1877.458391249" watchObservedRunningTime="2026-09-17 21:20:38.404203572 +0000 UTC m=+1878.098012896"
Sep 17 21:20:38.441283 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.441224    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjmwc\" (UniqueName: \"kubernetes.io/projected/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-api-access-gjmwc\") pod \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") "
Sep 17 21:20:38.441283 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.441262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls\") pod \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") "
Sep 17 21:20:38.441488 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.441310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-rbac-proxy-sar-config\") pod \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\" (UID: \"a9f46079-fee3-4582-9c6e-9b2e0403b0fa\") "
Sep 17 21:20:38.441762 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.441718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9f46079-fee3-4582-9c6e-9b2e0403b0fa" (UID: "a9f46079-fee3-4582-9c6e-9b2e0403b0fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:38.443739 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.443716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-api-access-gjmwc" (OuterVolumeSpecName: "kube-api-access-gjmwc") pod "a9f46079-fee3-4582-9c6e-9b2e0403b0fa" (UID: "a9f46079-fee3-4582-9c6e-9b2e0403b0fa"). InnerVolumeSpecName "kube-api-access-gjmwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:20:38.443808 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.443755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9f46079-fee3-4582-9c6e-9b2e0403b0fa" (UID: "a9f46079-fee3-4582-9c6e-9b2e0403b0fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:20:38.542528 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.542501    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:20:38.542528 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.542525    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjmwc\" (UniqueName: \"kubernetes.io/projected/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-kube-api-access-gjmwc\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:20:38.542702 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.542537    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f46079-fee3-4582-9c6e-9b2e0403b0fa-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:20:38.743640 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.743547    2579 generic.go:356] "Generic (PLEG): container finished" podID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerID="b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1" exitCode=0
Sep 17 21:20:38.743640 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.743607    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"
Sep 17 21:20:38.743856 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.743605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" event={"ID":"a9f46079-fee3-4582-9c6e-9b2e0403b0fa","Type":"ContainerDied","Data":"b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1"}
Sep 17 21:20:38.743856 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.743766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg" event={"ID":"a9f46079-fee3-4582-9c6e-9b2e0403b0fa","Type":"ContainerDied","Data":"8a5f5f491977a230d7b53bc609b080e3e0b2affd8ea0a2b65417ebf537e8dbd1"}
Sep 17 21:20:38.743856 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.743811    2579 scope.go:117] "RemoveContainer" containerID="afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063"
Sep 17 21:20:38.744139 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.744124    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:38.745820 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.745797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:20:38.752741 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.752724    2579 scope.go:117] "RemoveContainer" containerID="b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1"
Sep 17 21:20:38.760413 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.760397    2579 scope.go:117] "RemoveContainer" containerID="afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063"
Sep 17 21:20:38.760661 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:20:38.760641    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063\": container with ID starting with afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063 not found: ID does not exist" containerID="afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063"
Sep 17 21:20:38.760726 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.760667    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063"} err="failed to get container status \"afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063\": rpc error: code = NotFound desc = could not find container \"afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063\": container with ID starting with afffb5e16b76389778098fac0bfeb189be03b88a07c4e77b6f06ab573675e063 not found: ID does not exist"
Sep 17 21:20:38.760726 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.760683    2579 scope.go:117] "RemoveContainer" containerID="b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1"
Sep 17 21:20:38.760957 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:20:38.760940    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1\": container with ID starting with b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1 not found: ID does not exist" containerID="b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1"
Sep 17 21:20:38.760995 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.760965    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1"} err="failed to get container status \"b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1\": rpc error: code = NotFound desc = could not find container \"b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1\": container with ID starting with b38f7462f4ce87bb007206862cb20a771bf7968085cc592b6db1934c65f809a1 not found: ID does not exist"
Sep 17 21:20:38.778311 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.778290    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"]
Sep 17 21:20:38.782311 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:38.782287    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg"]
Sep 17 21:20:39.749236 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:39.749195    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:20:40.300713 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:40.300681    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" path="/var/lib/kubelet/pods/a9f46079-fee3-4582-9c6e-9b2e0403b0fa/volumes"
Sep 17 21:20:42.506484 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:42.506443    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:20:44.754827 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:44.754794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:20:44.755456 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:44.755237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:20:47.506248 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:47.506212    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:20:47.506723 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:47.506338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:20:52.505924 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:52.505886    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:20:54.755425 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:54.755389    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:20:57.505636 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:20:57.505595    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:02.506029 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:02.505991    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:04.755779 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:04.755744    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:21:04.872691 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:04.872660    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerID="3c9ce95817f49e50c6e67822e549a4b90cf372cde3b4bb2bc3d370aef135b292" exitCode=0
Sep 17 21:21:04.872794 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:04.872737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" event={"ID":"4a5933b7-d062-49ac-9557-f07dc1aa19b9","Type":"ContainerDied","Data":"3c9ce95817f49e50c6e67822e549a4b90cf372cde3b4bb2bc3d370aef135b292"}
Sep 17 21:21:04.991728 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:04.991702    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:21:05.066602 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.066536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls\") pod \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") "
Sep 17 21:21:05.066602 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.066567    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5933b7-d062-49ac-9557-f07dc1aa19b9-openshift-service-ca-bundle\") pod \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\" (UID: \"4a5933b7-d062-49ac-9557-f07dc1aa19b9\") "
Sep 17 21:21:05.066951 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.066927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5933b7-d062-49ac-9557-f07dc1aa19b9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4a5933b7-d062-49ac-9557-f07dc1aa19b9" (UID: "4a5933b7-d062-49ac-9557-f07dc1aa19b9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:21:05.068771 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.068749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a5933b7-d062-49ac-9557-f07dc1aa19b9" (UID: "4a5933b7-d062-49ac-9557-f07dc1aa19b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:21:05.167526 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.167499    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5933b7-d062-49ac-9557-f07dc1aa19b9-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:21:05.167526 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.167522    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a5933b7-d062-49ac-9557-f07dc1aa19b9-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:21:05.878282 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.878246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b" event={"ID":"4a5933b7-d062-49ac-9557-f07dc1aa19b9","Type":"ContainerDied","Data":"a415e5ece1ddd3f90924bb8373168da092ab6153e8886ae15e1035591ae9a156"}
Sep 17 21:21:05.878718 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.878308    2579 scope.go:117] "RemoveContainer" containerID="3c9ce95817f49e50c6e67822e549a4b90cf372cde3b4bb2bc3d370aef135b292"
Sep 17 21:21:05.878718 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.878265    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"
Sep 17 21:21:05.900691 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.900665    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"]
Sep 17 21:21:05.907348 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:05.907306    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b"]
Sep 17 21:21:06.299494 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:06.299418    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" path="/var/lib/kubelet/pods/4a5933b7-d062-49ac-9557-f07dc1aa19b9/volumes"
Sep 17 21:21:14.755973 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:14.755931    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:21:24.755390 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:24.755353    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:21:32.264412 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.264380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"]
Sep 17 21:21:32.264905 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.264596    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48" containerID="cri-o://b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428" gracePeriod=30
Sep 17 21:21:32.357822 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.357791    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"]
Sep 17 21:21:32.358089 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.358036    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" containerID="cri-o://ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e" gracePeriod=30
Sep 17 21:21:32.358235 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.358079    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kube-rbac-proxy" containerID="cri-o://2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859" gracePeriod=30
Sep 17 21:21:32.492766 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.492735    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"]
Sep 17 21:21:32.493159 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493142    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container"
Sep 17 21:21:32.493243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493162    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container"
Sep 17 21:21:32.493243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493188    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a"
Sep 17 21:21:32.493243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493197    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a"
Sep 17 21:21:32.493243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493209    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kube-rbac-proxy"
Sep 17 21:21:32.493243 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kube-rbac-proxy"
Sep 17 21:21:32.493525 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493304    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kube-rbac-proxy"
Sep 17 21:21:32.493525 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493341    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f46079-fee3-4582-9c6e-9b2e0403b0fa" containerName="kserve-container"
Sep 17 21:21:32.493525 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.493353    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5933b7-d062-49ac-9557-f07dc1aa19b9" containerName="ensemble-graph-31e6a"
Sep 17 21:21:32.496526 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.496506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.499336 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.499299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3ccd0-predictor-serving-cert\""
Sep 17 21:21:32.499620 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.499599    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3ccd0-kube-rbac-proxy-sar-config\""
Sep 17 21:21:32.507271 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.507248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"]
Sep 17 21:21:32.594367 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.594309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.594560 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.594451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xsqm\" (UniqueName: \"kubernetes.io/projected/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-api-access-9xsqm\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.594560 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.594502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ebad23-a56e-4e98-9080-9d3743cbc953-proxy-tls\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.695442 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.695409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.695613 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.695467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xsqm\" (UniqueName: \"kubernetes.io/projected/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-api-access-9xsqm\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.695613 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.695503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ebad23-a56e-4e98-9080-9d3743cbc953-proxy-tls\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.696182 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.696156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.698216 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.698189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ebad23-a56e-4e98-9080-9d3743cbc953-proxy-tls\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.705197 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.705176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xsqm\" (UniqueName: \"kubernetes.io/projected/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-api-access-9xsqm\") pod \"success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") " pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:32.808853 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:32.808821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:33.005456 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:33.005422    2579 generic.go:356] "Generic (PLEG): container finished" podID="509f7877-02a4-4acf-85ab-672c783f5118" containerID="2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859" exitCode=2
Sep 17 21:21:33.005661 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:33.005495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" event={"ID":"509f7877-02a4-4acf-85ab-672c783f5118","Type":"ContainerDied","Data":"2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859"}
Sep 17 21:21:33.082997 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:33.082973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"]
Sep 17 21:21:33.084950 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:21:33.084922    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ebad23_a56e_4e98_9080_9d3743cbc953.slice/crio-1ff1d1ff13369746676caee775e3b58e28c430cce07865296aff5e8d5f989416 WatchSource:0}: Error finding container 1ff1d1ff13369746676caee775e3b58e28c430cce07865296aff5e8d5f989416: Status 404 returned error can't find the container with id 1ff1d1ff13369746676caee775e3b58e28c430cce07865296aff5e8d5f989416
Sep 17 21:21:34.011622 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.011584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" event={"ID":"b1ebad23-a56e-4e98-9080-9d3743cbc953","Type":"ContainerStarted","Data":"5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2"}
Sep 17 21:21:34.012023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.011641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" event={"ID":"b1ebad23-a56e-4e98-9080-9d3743cbc953","Type":"ContainerStarted","Data":"169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd"}
Sep 17 21:21:34.012023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.011663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" event={"ID":"b1ebad23-a56e-4e98-9080-9d3743cbc953","Type":"ContainerStarted","Data":"1ff1d1ff13369746676caee775e3b58e28c430cce07865296aff5e8d5f989416"}
Sep 17 21:21:34.012023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.011685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:34.012023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.011830    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:34.013398 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.013371    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:21:34.033421 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.033370    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podStartSLOduration=2.033354343 podStartE2EDuration="2.033354343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:34.032591566 +0000 UTC m=+1933.726400916" watchObservedRunningTime="2026-09-17 21:21:34.033354343 +0000 UTC m=+1933.727163673"
Sep 17 21:21:34.441270 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.441234    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 17 21:21:34.446661 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.446640    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:21:34.756160 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.756073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:21:34.768470 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:34.768443    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:35.016436 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.016310    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:21:35.814915 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.814892    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:21:35.924023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.923942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wczdr\" (UniqueName: \"kubernetes.io/projected/509f7877-02a4-4acf-85ab-672c783f5118-kube-api-access-wczdr\") pod \"509f7877-02a4-4acf-85ab-672c783f5118\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") "
Sep 17 21:21:35.924023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.923984    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/509f7877-02a4-4acf-85ab-672c783f5118-proxy-tls\") pod \"509f7877-02a4-4acf-85ab-672c783f5118\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") "
Sep 17 21:21:35.924023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.924020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/509f7877-02a4-4acf-85ab-672c783f5118-kube-rbac-proxy-sar-config\") pod \"509f7877-02a4-4acf-85ab-672c783f5118\" (UID: \"509f7877-02a4-4acf-85ab-672c783f5118\") "
Sep 17 21:21:35.924474 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.924439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509f7877-02a4-4acf-85ab-672c783f5118-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "509f7877-02a4-4acf-85ab-672c783f5118" (UID: "509f7877-02a4-4acf-85ab-672c783f5118"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:21:35.926271 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.926248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509f7877-02a4-4acf-85ab-672c783f5118-kube-api-access-wczdr" (OuterVolumeSpecName: "kube-api-access-wczdr") pod "509f7877-02a4-4acf-85ab-672c783f5118" (UID: "509f7877-02a4-4acf-85ab-672c783f5118"). InnerVolumeSpecName "kube-api-access-wczdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:21:35.926377 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:35.926272    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f7877-02a4-4acf-85ab-672c783f5118-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "509f7877-02a4-4acf-85ab-672c783f5118" (UID: "509f7877-02a4-4acf-85ab-672c783f5118"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:21:36.022394 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.022361    2579 generic.go:356] "Generic (PLEG): container finished" podID="509f7877-02a4-4acf-85ab-672c783f5118" containerID="ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e" exitCode=0
Sep 17 21:21:36.022773 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.022435    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"
Sep 17 21:21:36.022773 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.022455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" event={"ID":"509f7877-02a4-4acf-85ab-672c783f5118","Type":"ContainerDied","Data":"ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e"}
Sep 17 21:21:36.022773 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.022529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd" event={"ID":"509f7877-02a4-4acf-85ab-672c783f5118","Type":"ContainerDied","Data":"c692c4f496f1041c5ffe7a664db29f33b5913e47bddbb43dda74a1b52777915d"}
Sep 17 21:21:36.022773 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.022566    2579 scope.go:117] "RemoveContainer" containerID="2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859"
Sep 17 21:21:36.024917 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.024889    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczdr\" (UniqueName: \"kubernetes.io/projected/509f7877-02a4-4acf-85ab-672c783f5118-kube-api-access-wczdr\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:21:36.024917 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.024917    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/509f7877-02a4-4acf-85ab-672c783f5118-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:21:36.025053 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.024932    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/509f7877-02a4-4acf-85ab-672c783f5118-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:21:36.035523 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.035508    2579 scope.go:117] "RemoveContainer" containerID="ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e"
Sep 17 21:21:36.042967 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.042948    2579 scope.go:117] "RemoveContainer" containerID="2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859"
Sep 17 21:21:36.043237 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:21:36.043220    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859\": container with ID starting with 2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859 not found: ID does not exist" containerID="2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859"
Sep 17 21:21:36.043292 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.043244    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859"} err="failed to get container status \"2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859\": rpc error: code = NotFound desc = could not find container \"2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859\": container with ID starting with 2dd3436e96f1036c2908faa555feca69cd06cd2227e56abb5bc0531e892b2859 not found: ID does not exist"
Sep 17 21:21:36.043292 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.043258    2579 scope.go:117] "RemoveContainer" containerID="ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e"
Sep 17 21:21:36.043491 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:21:36.043476    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e\": container with ID starting with ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e not found: ID does not exist" containerID="ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e"
Sep 17 21:21:36.043532 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.043496    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e"} err="failed to get container status \"ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e\": rpc error: code = NotFound desc = could not find container \"ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e\": container with ID starting with ea64dc3ffefa4eb7c80745e3f145f4582ba37158330e5dd6e33124a38961361e not found: ID does not exist"
Sep 17 21:21:36.048590 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.048568    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"]
Sep 17 21:21:36.053385 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.053344    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd"]
Sep 17 21:21:36.299737 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:36.299661    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="509f7877-02a4-4acf-85ab-672c783f5118" path="/var/lib/kubelet/pods/509f7877-02a4-4acf-85ab-672c783f5118/volumes"
Sep 17 21:21:39.768658 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:39.768622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:40.020736 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:40.020649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:21:40.021145 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:40.021115    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:21:44.768088 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:44.768044    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:44.768502 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:44.768169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:21:45.197853 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.197821    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"]
Sep 17 21:21:45.198168 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.198156    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container"
Sep 17 21:21:45.198219 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.198170    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container"
Sep 17 21:21:45.198219 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.198194    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kube-rbac-proxy"
Sep 17 21:21:45.198219 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.198200    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kube-rbac-proxy"
Sep 17 21:21:45.198313 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.198252    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kserve-container"
Sep 17 21:21:45.198313 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.198261    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f7877-02a4-4acf-85ab-672c783f5118" containerName="kube-rbac-proxy"
Sep 17 21:21:45.201186 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.201170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.204813 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.204790    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d3600-kube-rbac-proxy-sar-config\""
Sep 17 21:21:45.204952 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.204791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d3600-serving-cert\""
Sep 17 21:21:45.213415 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.213391    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"]
Sep 17 21:21:45.304358 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.304316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a482bba-1e70-4f49-a0d1-c3da5657ce67-proxy-tls\") pod \"splitter-graph-d3600-78645b4dcf-b5xv7\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") " pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.304513 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.304428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a482bba-1e70-4f49-a0d1-c3da5657ce67-openshift-service-ca-bundle\") pod \"splitter-graph-d3600-78645b4dcf-b5xv7\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") " pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.405193 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.405162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a482bba-1e70-4f49-a0d1-c3da5657ce67-proxy-tls\") pod \"splitter-graph-d3600-78645b4dcf-b5xv7\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") " pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.405369 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.405214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a482bba-1e70-4f49-a0d1-c3da5657ce67-openshift-service-ca-bundle\") pod \"splitter-graph-d3600-78645b4dcf-b5xv7\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") " pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.405805 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.405785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a482bba-1e70-4f49-a0d1-c3da5657ce67-openshift-service-ca-bundle\") pod \"splitter-graph-d3600-78645b4dcf-b5xv7\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") " pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.407719 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.407702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a482bba-1e70-4f49-a0d1-c3da5657ce67-proxy-tls\") pod \"splitter-graph-d3600-78645b4dcf-b5xv7\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") " pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.511915 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.511828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:45.791958 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:45.791871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"]
Sep 17 21:21:45.799018 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:21:45.798986    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a482bba_1e70_4f49_a0d1_c3da5657ce67.slice/crio-be904fca1fd75b03a1a46a73f7b1f9d1e36f41597707e0bc1b560a1cab04cd72 WatchSource:0}: Error finding container be904fca1fd75b03a1a46a73f7b1f9d1e36f41597707e0bc1b560a1cab04cd72: Status 404 returned error can't find the container with id be904fca1fd75b03a1a46a73f7b1f9d1e36f41597707e0bc1b560a1cab04cd72
Sep 17 21:21:46.072229 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:46.072138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" event={"ID":"5a482bba-1e70-4f49-a0d1-c3da5657ce67","Type":"ContainerStarted","Data":"08cdadfdb1cb1bb3d3e9bd671e1f578d352d9ab724883b1600b627149d6b3296"}
Sep 17 21:21:46.072229 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:46.072197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" event={"ID":"5a482bba-1e70-4f49-a0d1-c3da5657ce67","Type":"ContainerStarted","Data":"be904fca1fd75b03a1a46a73f7b1f9d1e36f41597707e0bc1b560a1cab04cd72"}
Sep 17 21:21:46.072428 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:46.072264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:46.092639 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:46.092581    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podStartSLOduration=1.092562355 podStartE2EDuration="1.092562355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:46.092355886 +0000 UTC m=+1945.786165238" watchObservedRunningTime="2026-09-17 21:21:46.092562355 +0000 UTC m=+1945.786371687"
Sep 17 21:21:49.768647 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:49.768608    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:50.021276 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:50.021184    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:21:52.081956 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:52.081927    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:21:54.768000 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:54.767955    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:56.194748 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.194713    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"]
Sep 17 21:21:56.195182 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.194928    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600" containerID="cri-o://08cdadfdb1cb1bb3d3e9bd671e1f578d352d9ab724883b1600b627149d6b3296" gracePeriod=30
Sep 17 21:21:56.333075 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.333040    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"]
Sep 17 21:21:56.333312 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.333288    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container" containerID="cri-o://09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712" gracePeriod=30
Sep 17 21:21:56.333429 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.333400    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kube-rbac-proxy" containerID="cri-o://0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560" gracePeriod=30
Sep 17 21:21:56.483963 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.483926    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"]
Sep 17 21:21:56.487282 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.487259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.491194 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.491173    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1ff2-predictor-serving-cert\""
Sep 17 21:21:56.491314 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.491244    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1ff2-kube-rbac-proxy-sar-config\""
Sep 17 21:21:56.505516 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.505491    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"]
Sep 17 21:21:56.595531 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.595495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlz6h\" (UniqueName: \"kubernetes.io/projected/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-api-access-jlz6h\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.595728 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.595535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.595728 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.595669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.696963 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.696928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlz6h\" (UniqueName: \"kubernetes.io/projected/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-api-access-jlz6h\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.697149 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.696972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.697149 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.697044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.697290 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:21:56.697166    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-serving-cert: secret "success-200-isvc-a1ff2-predictor-serving-cert" not found
Sep 17 21:21:56.697290 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:21:56.697257    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls podName:66ac1b73-0fa6-4b81-8750-cabf29254126 nodeName:}" failed. No retries permitted until 2026-09-17 21:21:57.197234794 +0000 UTC m=+1956.891044321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls") pod "success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" (UID: "66ac1b73-0fa6-4b81-8750-cabf29254126") : secret "success-200-isvc-a1ff2-predictor-serving-cert" not found
Sep 17 21:21:56.698009 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.697987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:56.708839 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:56.708782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlz6h\" (UniqueName: \"kubernetes.io/projected/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-api-access-jlz6h\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:57.080969 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:57.080938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:57.122958 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:57.122926    2579 generic.go:356] "Generic (PLEG): container finished" podID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerID="0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560" exitCode=2
Sep 17 21:21:57.123097 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:57.122996    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" event={"ID":"1941a697-469b-4130-8b62-e7fbd3ef0eed","Type":"ContainerDied","Data":"0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560"}
Sep 17 21:21:57.202330 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:57.202298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:57.204808 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:57.204780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls\") pod \"success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") " pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:57.400755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:57.400669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:57.677828 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:57.677799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"]
Sep 17 21:21:57.679433 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:21:57.679408    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ac1b73_0fa6_4b81_8750_cabf29254126.slice/crio-9b069c92c9def09a882cf88194a03681e187959cb9be984a0989f8fe569b712d WatchSource:0}: Error finding container 9b069c92c9def09a882cf88194a03681e187959cb9be984a0989f8fe569b712d: Status 404 returned error can't find the container with id 9b069c92c9def09a882cf88194a03681e187959cb9be984a0989f8fe569b712d
Sep 17 21:21:58.129493 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:58.129455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" event={"ID":"66ac1b73-0fa6-4b81-8750-cabf29254126","Type":"ContainerStarted","Data":"878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de"}
Sep 17 21:21:58.129672 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:58.129558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:58.129672 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:58.129589    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:21:58.129672 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:58.129616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" event={"ID":"66ac1b73-0fa6-4b81-8750-cabf29254126","Type":"ContainerStarted","Data":"71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54"}
Sep 17 21:21:58.129672 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:58.129650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" event={"ID":"66ac1b73-0fa6-4b81-8750-cabf29254126","Type":"ContainerStarted","Data":"9b069c92c9def09a882cf88194a03681e187959cb9be984a0989f8fe569b712d"}
Sep 17 21:21:58.131185 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:58.131155    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:58.151004 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:58.150949    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podStartSLOduration=2.150931851 podStartE2EDuration="2.150931851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:58.150237855 +0000 UTC m=+1957.844047210" watchObservedRunningTime="2026-09-17 21:21:58.150931851 +0000 UTC m=+1957.844741183"
Sep 17 21:21:59.134669 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:59.134632    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:59.749964 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:59.749921    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 21:21:59.768830 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:59.768799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:21:59.982666 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:21:59.982644    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:22:00.021129 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.021046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:22:00.026384 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.026362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls\") pod \"1941a697-469b-4130-8b62-e7fbd3ef0eed\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") "
Sep 17 21:22:00.026472 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.026415    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mcdb\" (UniqueName: \"kubernetes.io/projected/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-api-access-4mcdb\") pod \"1941a697-469b-4130-8b62-e7fbd3ef0eed\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") "
Sep 17 21:22:00.026472 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.026448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-rbac-proxy-sar-config\") pod \"1941a697-469b-4130-8b62-e7fbd3ef0eed\" (UID: \"1941a697-469b-4130-8b62-e7fbd3ef0eed\") "
Sep 17 21:22:00.026828 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.026803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1941a697-469b-4130-8b62-e7fbd3ef0eed" (UID: "1941a697-469b-4130-8b62-e7fbd3ef0eed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:00.028680 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.028656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-api-access-4mcdb" (OuterVolumeSpecName: "kube-api-access-4mcdb") pod "1941a697-469b-4130-8b62-e7fbd3ef0eed" (UID: "1941a697-469b-4130-8b62-e7fbd3ef0eed"). InnerVolumeSpecName "kube-api-access-4mcdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:22:00.028788 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.028700    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1941a697-469b-4130-8b62-e7fbd3ef0eed" (UID: "1941a697-469b-4130-8b62-e7fbd3ef0eed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:22:00.127778 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.127735    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mcdb\" (UniqueName: \"kubernetes.io/projected/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-api-access-4mcdb\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:22:00.127778 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.127770    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1941a697-469b-4130-8b62-e7fbd3ef0eed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:22:00.127778 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.127780    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1941a697-469b-4130-8b62-e7fbd3ef0eed-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:22:00.140062 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.140033    2579 generic.go:356] "Generic (PLEG): container finished" podID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerID="09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712" exitCode=0
Sep 17 21:22:00.140427 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.140088    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"
Sep 17 21:22:00.140427 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.140093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" event={"ID":"1941a697-469b-4130-8b62-e7fbd3ef0eed","Type":"ContainerDied","Data":"09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712"}
Sep 17 21:22:00.140427 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.140156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw" event={"ID":"1941a697-469b-4130-8b62-e7fbd3ef0eed","Type":"ContainerDied","Data":"3851cd788ba3630bfddda0681faf4eff38a010102b49c88d1659431d1fac630f"}
Sep 17 21:22:00.140427 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.140192    2579 scope.go:117] "RemoveContainer" containerID="0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560"
Sep 17 21:22:00.149411 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.149392    2579 scope.go:117] "RemoveContainer" containerID="09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712"
Sep 17 21:22:00.156975 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.156960    2579 scope.go:117] "RemoveContainer" containerID="0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560"
Sep 17 21:22:00.157233 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:22:00.157215    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560\": container with ID starting with 0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560 not found: ID does not exist" containerID="0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560"
Sep 17 21:22:00.157300 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.157241    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560"} err="failed to get container status \"0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560\": rpc error: code = NotFound desc = could not find container \"0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560\": container with ID starting with 0aa4ba84abebd861c61c08aa1a18240c9674cc492eb46a9c4bed08bea424e560 not found: ID does not exist"
Sep 17 21:22:00.157300 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.157258    2579 scope.go:117] "RemoveContainer" containerID="09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712"
Sep 17 21:22:00.157537 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:22:00.157519    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712\": container with ID starting with 09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712 not found: ID does not exist" containerID="09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712"
Sep 17 21:22:00.157579 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.157542    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712"} err="failed to get container status \"09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712\": rpc error: code = NotFound desc = could not find container \"09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712\": container with ID starting with 09341d71568d167148f877764ba1f8e3d822b3526f232c20d9b3cf07a12eb712 not found: ID does not exist"
Sep 17 21:22:00.162874 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.162853    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"]
Sep 17 21:22:00.167091 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.167071    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw"]
Sep 17 21:22:00.299701 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:00.299617    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" path="/var/lib/kubelet/pods/1941a697-469b-4130-8b62-e7fbd3ef0eed/volumes"
Sep 17 21:22:02.080724 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.080688    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:22:02.416901 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.416874    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:22:02.447647 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.447622    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-openshift-service-ca-bundle\") pod \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") "
Sep 17 21:22:02.447772 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.447731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-proxy-tls\") pod \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\" (UID: \"27d086a6-ed2b-4a98-a64d-a49808f7e6cd\") "
Sep 17 21:22:02.447983 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.447958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "27d086a6-ed2b-4a98-a64d-a49808f7e6cd" (UID: "27d086a6-ed2b-4a98-a64d-a49808f7e6cd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:02.450167 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.450141    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27d086a6-ed2b-4a98-a64d-a49808f7e6cd" (UID: "27d086a6-ed2b-4a98-a64d-a49808f7e6cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:22:02.549165 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.549132    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:22:02.549165 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:02.549163    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d086a6-ed2b-4a98-a64d-a49808f7e6cd-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:22:03.154675 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.154643    2579 generic.go:356] "Generic (PLEG): container finished" podID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerID="b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428" exitCode=0
Sep 17 21:22:03.155176 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.154686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" event={"ID":"27d086a6-ed2b-4a98-a64d-a49808f7e6cd","Type":"ContainerDied","Data":"b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428"}
Sep 17 21:22:03.155176 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.154698    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"
Sep 17 21:22:03.155176 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.154752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4" event={"ID":"27d086a6-ed2b-4a98-a64d-a49808f7e6cd","Type":"ContainerDied","Data":"ade226ede50b23ac0497cb52a6398e9a309d2815279cd54afa17e7dd8d231e15"}
Sep 17 21:22:03.155176 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.154773    2579 scope.go:117] "RemoveContainer" containerID="b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428"
Sep 17 21:22:03.163208 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.163192    2579 scope.go:117] "RemoveContainer" containerID="b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428"
Sep 17 21:22:03.163503 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:22:03.163483    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428\": container with ID starting with b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428 not found: ID does not exist" containerID="b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428"
Sep 17 21:22:03.163548 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.163513    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428"} err="failed to get container status \"b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428\": rpc error: code = NotFound desc = could not find container \"b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428\": container with ID starting with b2a1744481a434f7796bbea79816617c9dc6598c78588773cdcc611ea1a0a428 not found: ID does not exist"
Sep 17 21:22:03.175066 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.175043    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"]
Sep 17 21:22:03.177251 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:03.177231    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4"]
Sep 17 21:22:04.139572 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:04.139544    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:22:04.140143 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:04.140118    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:04.299294 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:04.299262    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" path="/var/lib/kubelet/pods/27d086a6-ed2b-4a98-a64d-a49808f7e6cd/volumes"
Sep 17 21:22:07.081293 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:07.081254    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:22:07.081710 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:07.081376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:22:10.022157 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:10.022117    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:22:12.081598 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:12.081559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:22:14.140311 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:14.140270    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:17.080971 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:17.080919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:22:20.021431 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:20.021392    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:22:22.080701 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:22.080654    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:22:24.140256 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:24.140211    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:26.268449 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.268420    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerID="08cdadfdb1cb1bb3d3e9bd671e1f578d352d9ab724883b1600b627149d6b3296" exitCode=0
Sep 17 21:22:26.268796 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.268480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" event={"ID":"5a482bba-1e70-4f49-a0d1-c3da5657ce67","Type":"ContainerDied","Data":"08cdadfdb1cb1bb3d3e9bd671e1f578d352d9ab724883b1600b627149d6b3296"}
Sep 17 21:22:26.349678 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.349649    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:22:26.450112 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.450019    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a482bba-1e70-4f49-a0d1-c3da5657ce67-proxy-tls\") pod \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") "
Sep 17 21:22:26.450112 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.450109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a482bba-1e70-4f49-a0d1-c3da5657ce67-openshift-service-ca-bundle\") pod \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\" (UID: \"5a482bba-1e70-4f49-a0d1-c3da5657ce67\") "
Sep 17 21:22:26.450541 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.450511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a482bba-1e70-4f49-a0d1-c3da5657ce67-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5a482bba-1e70-4f49-a0d1-c3da5657ce67" (UID: "5a482bba-1e70-4f49-a0d1-c3da5657ce67"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:26.452294 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.452266    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a482bba-1e70-4f49-a0d1-c3da5657ce67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a482bba-1e70-4f49-a0d1-c3da5657ce67" (UID: "5a482bba-1e70-4f49-a0d1-c3da5657ce67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:22:26.551111 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.551077    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a482bba-1e70-4f49-a0d1-c3da5657ce67-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:22:26.551111 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:26.551107    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a482bba-1e70-4f49-a0d1-c3da5657ce67-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:22:27.274219 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:27.274187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"
Sep 17 21:22:27.274219 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:27.274197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7" event={"ID":"5a482bba-1e70-4f49-a0d1-c3da5657ce67","Type":"ContainerDied","Data":"be904fca1fd75b03a1a46a73f7b1f9d1e36f41597707e0bc1b560a1cab04cd72"}
Sep 17 21:22:27.274752 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:27.274261    2579 scope.go:117] "RemoveContainer" containerID="08cdadfdb1cb1bb3d3e9bd671e1f578d352d9ab724883b1600b627149d6b3296"
Sep 17 21:22:27.297525 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:27.297501    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"]
Sep 17 21:22:27.303577 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:27.303550    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7"]
Sep 17 21:22:28.299053 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:28.299019    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" path="/var/lib/kubelet/pods/5a482bba-1e70-4f49-a0d1-c3da5657ce67/volumes"
Sep 17 21:22:30.022411 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:30.022383    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:22:34.140719 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:34.140679    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:42.534975 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.534935    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"]
Sep 17 21:22:42.536204 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536178    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48"
Sep 17 21:22:42.536276 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536209    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48"
Sep 17 21:22:42.536276 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536224    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container"
Sep 17 21:22:42.536276 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536242    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container"
Sep 17 21:22:42.536276 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536252    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600"
Sep 17 21:22:42.536276 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536260    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600"
Sep 17 21:22:42.536564 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536287    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kube-rbac-proxy"
Sep 17 21:22:42.536564 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536297    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kube-rbac-proxy"
Sep 17 21:22:42.536564 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536487    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kube-rbac-proxy"
Sep 17 21:22:42.536564 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536503    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a482bba-1e70-4f49-a0d1-c3da5657ce67" containerName="splitter-graph-d3600"
Sep 17 21:22:42.536564 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536521    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d086a6-ed2b-4a98-a64d-a49808f7e6cd" containerName="sequence-graph-b5a48"
Sep 17 21:22:42.536564 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.536532    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1941a697-469b-4130-8b62-e7fbd3ef0eed" containerName="kserve-container"
Sep 17 21:22:42.543182 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.542773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:42.547155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.547130    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3ccd0-serving-cert\""
Sep 17 21:22:42.547339 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.547134    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3ccd0-kube-rbac-proxy-sar-config\""
Sep 17 21:22:42.548334 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.548299    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"]
Sep 17 21:22:42.685087 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.685052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b61a42-da19-4b59-9933-16acdef775f0-proxy-tls\") pod \"switch-graph-3ccd0-6464d894c-mplj9\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") " pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:42.685276 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.685149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b61a42-da19-4b59-9933-16acdef775f0-openshift-service-ca-bundle\") pod \"switch-graph-3ccd0-6464d894c-mplj9\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") " pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:42.785614 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.785541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b61a42-da19-4b59-9933-16acdef775f0-proxy-tls\") pod \"switch-graph-3ccd0-6464d894c-mplj9\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") " pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:42.785614 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.785608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b61a42-da19-4b59-9933-16acdef775f0-openshift-service-ca-bundle\") pod \"switch-graph-3ccd0-6464d894c-mplj9\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") " pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:42.786179 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.786161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b61a42-da19-4b59-9933-16acdef775f0-openshift-service-ca-bundle\") pod \"switch-graph-3ccd0-6464d894c-mplj9\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") " pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:42.788098 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.788073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b61a42-da19-4b59-9933-16acdef775f0-proxy-tls\") pod \"switch-graph-3ccd0-6464d894c-mplj9\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") " pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:42.856714 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:42.856688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:43.127412 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:43.127389    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"]
Sep 17 21:22:43.129504 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:22:43.129479    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b61a42_da19_4b59_9933_16acdef775f0.slice/crio-b9bdb6770ae2a8cb69a05e547e7d0d4de24a5909f5560e656ae33c19b531b1ab WatchSource:0}: Error finding container b9bdb6770ae2a8cb69a05e547e7d0d4de24a5909f5560e656ae33c19b531b1ab: Status 404 returned error can't find the container with id b9bdb6770ae2a8cb69a05e547e7d0d4de24a5909f5560e656ae33c19b531b1ab
Sep 17 21:22:43.347023 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:43.346982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" event={"ID":"a6b61a42-da19-4b59-9933-16acdef775f0","Type":"ContainerStarted","Data":"2a807dea20fee82e89f39fd6d4a754880e36fcac537d35e8c2a4b5cf81fd9041"}
Sep 17 21:22:43.347195 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:43.347040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" event={"ID":"a6b61a42-da19-4b59-9933-16acdef775f0","Type":"ContainerStarted","Data":"b9bdb6770ae2a8cb69a05e547e7d0d4de24a5909f5560e656ae33c19b531b1ab"}
Sep 17 21:22:43.347195 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:43.347084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:43.366096 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:43.366045    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podStartSLOduration=1.3660298929999999 podStartE2EDuration="1.366029893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:22:43.363979687 +0000 UTC m=+2003.057789018" watchObservedRunningTime="2026-09-17 21:22:43.366029893 +0000 UTC m=+2003.059839224"
Sep 17 21:22:44.140155 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:44.140121    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:49.357758 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:49.357723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:22:54.141512 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:22:54.141479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:23:06.440843 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.440813    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"]
Sep 17 21:23:06.444417 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.444392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:06.446573 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.446552    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a1ff2-serving-cert\""
Sep 17 21:23:06.446573 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.446560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a1ff2-kube-rbac-proxy-sar-config\""
Sep 17 21:23:06.453776 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.453750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"]
Sep 17 21:23:06.484682 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.484654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls\") pod \"splitter-graph-a1ff2-574bdd87b5-hktnd\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") " pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:06.484833 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.484704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695662f2-6e5c-40ed-a5af-9ccc683f51bb-openshift-service-ca-bundle\") pod \"splitter-graph-a1ff2-574bdd87b5-hktnd\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") " pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:06.585244 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.585210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695662f2-6e5c-40ed-a5af-9ccc683f51bb-openshift-service-ca-bundle\") pod \"splitter-graph-a1ff2-574bdd87b5-hktnd\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") " pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:06.585425 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.585305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls\") pod \"splitter-graph-a1ff2-574bdd87b5-hktnd\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") " pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:06.585476 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:23:06.585450    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-a1ff2-serving-cert: secret "splitter-graph-a1ff2-serving-cert" not found
Sep 17 21:23:06.585537 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:23:06.585526    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls podName:695662f2-6e5c-40ed-a5af-9ccc683f51bb nodeName:}" failed. No retries permitted until 2026-09-17 21:23:07.085509838 +0000 UTC m=+2026.779319148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls") pod "splitter-graph-a1ff2-574bdd87b5-hktnd" (UID: "695662f2-6e5c-40ed-a5af-9ccc683f51bb") : secret "splitter-graph-a1ff2-serving-cert" not found
Sep 17 21:23:06.585909 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:06.585886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695662f2-6e5c-40ed-a5af-9ccc683f51bb-openshift-service-ca-bundle\") pod \"splitter-graph-a1ff2-574bdd87b5-hktnd\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") " pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:07.089339 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:07.089280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls\") pod \"splitter-graph-a1ff2-574bdd87b5-hktnd\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") " pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:07.091829 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:07.091808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls\") pod \"splitter-graph-a1ff2-574bdd87b5-hktnd\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") " pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:07.357251 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:07.357151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:07.627649 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:07.627621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"]
Sep 17 21:23:07.630422 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:23:07.630395    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695662f2_6e5c_40ed_a5af_9ccc683f51bb.slice/crio-2958d000e724a79d7ad5ddac7400fcd8c20a60012eda1c110824b25cb6c573d8 WatchSource:0}: Error finding container 2958d000e724a79d7ad5ddac7400fcd8c20a60012eda1c110824b25cb6c573d8: Status 404 returned error can't find the container with id 2958d000e724a79d7ad5ddac7400fcd8c20a60012eda1c110824b25cb6c573d8
Sep 17 21:23:08.457621 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:08.457585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" event={"ID":"695662f2-6e5c-40ed-a5af-9ccc683f51bb","Type":"ContainerStarted","Data":"3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16"}
Sep 17 21:23:08.457798 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:08.457656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:23:08.457798 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:08.457680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" event={"ID":"695662f2-6e5c-40ed-a5af-9ccc683f51bb","Type":"ContainerStarted","Data":"2958d000e724a79d7ad5ddac7400fcd8c20a60012eda1c110824b25cb6c573d8"}
Sep 17 21:23:08.476241 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:08.476201    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podStartSLOduration=2.476186776 podStartE2EDuration="2.476186776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:23:08.474345482 +0000 UTC m=+2028.168154834" watchObservedRunningTime="2026-09-17 21:23:08.476186776 +0000 UTC m=+2028.169996139"
Sep 17 21:23:14.467139 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:23:14.467104    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:24:22.392613 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:24:22.392583    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:24:22.396156 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:24:22.396124    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:24:22.397290 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:24:22.397273    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:24:22.400691 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:24:22.400673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:29:22.422911 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:29:22.422888    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:29:22.425927 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:29:22.425905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:29:22.427755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:29:22.427739    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:29:22.430608 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:29:22.430589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:31:21.623729 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:21.623695    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"]
Sep 17 21:31:21.624429 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:21.623950    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2" containerID="cri-o://3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16" gracePeriod=30
Sep 17 21:31:21.787935 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:21.787895    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"]
Sep 17 21:31:21.788253 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:21.788180    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" containerID="cri-o://71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54" gracePeriod=30
Sep 17 21:31:21.788253 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:21.788213    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kube-rbac-proxy" containerID="cri-o://878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de" gracePeriod=30
Sep 17 21:31:22.675004 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:22.674963    2579 generic.go:356] "Generic (PLEG): container finished" podID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerID="878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de" exitCode=2
Sep 17 21:31:22.675400 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:22.675042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" event={"ID":"66ac1b73-0fa6-4b81-8750-cabf29254126","Type":"ContainerDied","Data":"878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de"}
Sep 17 21:31:24.135046 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:24.135008    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 21:31:24.140369 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:24.140347    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:31:24.466044 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:24.465963    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:31:25.036018 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.035995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:31:25.092182 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.092154    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-rbac-proxy-sar-config\") pod \"66ac1b73-0fa6-4b81-8750-cabf29254126\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") "
Sep 17 21:31:25.092315 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.092198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlz6h\" (UniqueName: \"kubernetes.io/projected/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-api-access-jlz6h\") pod \"66ac1b73-0fa6-4b81-8750-cabf29254126\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") "
Sep 17 21:31:25.092315 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.092214    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls\") pod \"66ac1b73-0fa6-4b81-8750-cabf29254126\" (UID: \"66ac1b73-0fa6-4b81-8750-cabf29254126\") "
Sep 17 21:31:25.092593 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.092565    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66ac1b73-0fa6-4b81-8750-cabf29254126" (UID: "66ac1b73-0fa6-4b81-8750-cabf29254126"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:31:25.094428 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.094400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66ac1b73-0fa6-4b81-8750-cabf29254126" (UID: "66ac1b73-0fa6-4b81-8750-cabf29254126"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:31:25.094533 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.094454    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-api-access-jlz6h" (OuterVolumeSpecName: "kube-api-access-jlz6h") pod "66ac1b73-0fa6-4b81-8750-cabf29254126" (UID: "66ac1b73-0fa6-4b81-8750-cabf29254126"). InnerVolumeSpecName "kube-api-access-jlz6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:31:25.193343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.193256    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:31:25.193343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.193286    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlz6h\" (UniqueName: \"kubernetes.io/projected/66ac1b73-0fa6-4b81-8750-cabf29254126-kube-api-access-jlz6h\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:31:25.193343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.193296    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66ac1b73-0fa6-4b81-8750-cabf29254126-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:31:25.689777 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.689739    2579 generic.go:356] "Generic (PLEG): container finished" podID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerID="71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54" exitCode=0
Sep 17 21:31:25.689961 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.689786    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"
Sep 17 21:31:25.689961 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.689802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" event={"ID":"66ac1b73-0fa6-4b81-8750-cabf29254126","Type":"ContainerDied","Data":"71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54"}
Sep 17 21:31:25.689961 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.689846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v" event={"ID":"66ac1b73-0fa6-4b81-8750-cabf29254126","Type":"ContainerDied","Data":"9b069c92c9def09a882cf88194a03681e187959cb9be984a0989f8fe569b712d"}
Sep 17 21:31:25.689961 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.689869    2579 scope.go:117] "RemoveContainer" containerID="878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de"
Sep 17 21:31:25.699437 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.699419    2579 scope.go:117] "RemoveContainer" containerID="71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54"
Sep 17 21:31:25.706601 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.706585    2579 scope.go:117] "RemoveContainer" containerID="878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de"
Sep 17 21:31:25.706861 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:31:25.706843    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de\": container with ID starting with 878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de not found: ID does not exist" containerID="878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de"
Sep 17 21:31:25.706907 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.706870    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de"} err="failed to get container status \"878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de\": rpc error: code = NotFound desc = could not find container \"878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de\": container with ID starting with 878fb56cc382ce04013bc5edfa370781c66c2c93fd2f18809236e05e3c2c67de not found: ID does not exist"
Sep 17 21:31:25.706907 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.706886    2579 scope.go:117] "RemoveContainer" containerID="71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54"
Sep 17 21:31:25.707113 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:31:25.707094    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54\": container with ID starting with 71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54 not found: ID does not exist" containerID="71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54"
Sep 17 21:31:25.707159 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.707119    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54"} err="failed to get container status \"71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54\": rpc error: code = NotFound desc = could not find container \"71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54\": container with ID starting with 71f91330a08ad9df41e0e0a0b5c5fca4e6028a7c20edc31c22b40979bdb45d54 not found: ID does not exist"
Sep 17 21:31:25.714176 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.714156    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"]
Sep 17 21:31:25.717951 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:25.717934    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v"]
Sep 17 21:31:26.300285 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:26.300252    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" path="/var/lib/kubelet/pods/66ac1b73-0fa6-4b81-8750-cabf29254126/volumes"
Sep 17 21:31:29.465203 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:29.465159    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:31:34.465644 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:34.465600    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:31:34.466041 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:34.465703    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:31:39.465416 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:39.465373    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:31:44.465471 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:44.465429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:31:49.465561 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:49.465527    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:31:51.770625 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.770601    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:31:51.806800 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.806772    2579 generic.go:356] "Generic (PLEG): container finished" podID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerID="3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16" exitCode=0
Sep 17 21:31:51.806977 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.806816    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"
Sep 17 21:31:51.806977 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.806827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" event={"ID":"695662f2-6e5c-40ed-a5af-9ccc683f51bb","Type":"ContainerDied","Data":"3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16"}
Sep 17 21:31:51.806977 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.806871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd" event={"ID":"695662f2-6e5c-40ed-a5af-9ccc683f51bb","Type":"ContainerDied","Data":"2958d000e724a79d7ad5ddac7400fcd8c20a60012eda1c110824b25cb6c573d8"}
Sep 17 21:31:51.806977 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.806891    2579 scope.go:117] "RemoveContainer" containerID="3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16"
Sep 17 21:31:51.815086 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.815069    2579 scope.go:117] "RemoveContainer" containerID="3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16"
Sep 17 21:31:51.815395 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:31:51.815373    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16\": container with ID starting with 3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16 not found: ID does not exist" containerID="3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16"
Sep 17 21:31:51.815486 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.815402    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16"} err="failed to get container status \"3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16\": rpc error: code = NotFound desc = could not find container \"3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16\": container with ID starting with 3709a55c5c9f2de1293b585472c9f05e76ba2b15345c5d6ac4a9e5343405ac16 not found: ID does not exist"
Sep 17 21:31:51.910907 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.910833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls\") pod \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") "
Sep 17 21:31:51.911041 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.910909    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695662f2-6e5c-40ed-a5af-9ccc683f51bb-openshift-service-ca-bundle\") pod \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\" (UID: \"695662f2-6e5c-40ed-a5af-9ccc683f51bb\") "
Sep 17 21:31:51.911208 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.911186    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695662f2-6e5c-40ed-a5af-9ccc683f51bb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "695662f2-6e5c-40ed-a5af-9ccc683f51bb" (UID: "695662f2-6e5c-40ed-a5af-9ccc683f51bb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:31:51.913029 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:51.912999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "695662f2-6e5c-40ed-a5af-9ccc683f51bb" (UID: "695662f2-6e5c-40ed-a5af-9ccc683f51bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:31:52.011511 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:52.011486    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695662f2-6e5c-40ed-a5af-9ccc683f51bb-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:31:52.011511 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:52.011507    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695662f2-6e5c-40ed-a5af-9ccc683f51bb-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:31:52.129382 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:52.129356    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"]
Sep 17 21:31:52.134955 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:52.134935    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd"]
Sep 17 21:31:52.300105 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:31:52.300034    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" path="/var/lib/kubelet/pods/695662f2-6e5c-40ed-a5af-9ccc683f51bb/volumes"
Sep 17 21:34:22.450344 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:34:22.450299    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:34:22.453962 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:34:22.453942    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:34:22.455210 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:34:22.455181    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:34:22.458873 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:34:22.458854    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:39:02.727516 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:02.727438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"]
Sep 17 21:39:02.728033 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:02.727674    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0" containerID="cri-o://2a807dea20fee82e89f39fd6d4a754880e36fcac537d35e8c2a4b5cf81fd9041" gracePeriod=30
Sep 17 21:39:02.900863 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:02.900830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"]
Sep 17 21:39:02.901117 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:02.901066    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container" containerID="cri-o://169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd" gracePeriod=30
Sep 17 21:39:02.901257 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:02.901098    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kube-rbac-proxy" containerID="cri-o://5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2" gracePeriod=30
Sep 17 21:39:03.715741 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:03.715700    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerID="5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2" exitCode=2
Sep 17 21:39:03.715909 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:03.715773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" event={"ID":"b1ebad23-a56e-4e98-9080-9d3743cbc953","Type":"ContainerDied","Data":"5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2"}
Sep 17 21:39:04.355577 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:04.355542    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:39:05.017200 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:05.017154    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 21:39:06.151435 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.151412    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:39:06.215231 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.215198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-rbac-proxy-sar-config\") pod \"b1ebad23-a56e-4e98-9080-9d3743cbc953\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") "
Sep 17 21:39:06.215422 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.215288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xsqm\" (UniqueName: \"kubernetes.io/projected/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-api-access-9xsqm\") pod \"b1ebad23-a56e-4e98-9080-9d3743cbc953\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") "
Sep 17 21:39:06.215422 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.215351    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ebad23-a56e-4e98-9080-9d3743cbc953-proxy-tls\") pod \"b1ebad23-a56e-4e98-9080-9d3743cbc953\" (UID: \"b1ebad23-a56e-4e98-9080-9d3743cbc953\") "
Sep 17 21:39:06.215634 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.215601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1ebad23-a56e-4e98-9080-9d3743cbc953" (UID: "b1ebad23-a56e-4e98-9080-9d3743cbc953"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:39:06.217504 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.217481    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ebad23-a56e-4e98-9080-9d3743cbc953-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1ebad23-a56e-4e98-9080-9d3743cbc953" (UID: "b1ebad23-a56e-4e98-9080-9d3743cbc953"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:39:06.217610 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.217519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-api-access-9xsqm" (OuterVolumeSpecName: "kube-api-access-9xsqm") pod "b1ebad23-a56e-4e98-9080-9d3743cbc953" (UID: "b1ebad23-a56e-4e98-9080-9d3743cbc953"). InnerVolumeSpecName "kube-api-access-9xsqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:39:06.316085 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.316060    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:39:06.316085 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.316084    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xsqm\" (UniqueName: \"kubernetes.io/projected/b1ebad23-a56e-4e98-9080-9d3743cbc953-kube-api-access-9xsqm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:39:06.316235 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.316093    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ebad23-a56e-4e98-9080-9d3743cbc953-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:39:06.733417 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.733385    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerID="169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd" exitCode=0
Sep 17 21:39:06.733600 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.733430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" event={"ID":"b1ebad23-a56e-4e98-9080-9d3743cbc953","Type":"ContainerDied","Data":"169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd"}
Sep 17 21:39:06.733600 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.733439    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"
Sep 17 21:39:06.733600 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.733477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg" event={"ID":"b1ebad23-a56e-4e98-9080-9d3743cbc953","Type":"ContainerDied","Data":"1ff1d1ff13369746676caee775e3b58e28c430cce07865296aff5e8d5f989416"}
Sep 17 21:39:06.733600 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.733513    2579 scope.go:117] "RemoveContainer" containerID="5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2"
Sep 17 21:39:06.741868 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.741850    2579 scope.go:117] "RemoveContainer" containerID="169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd"
Sep 17 21:39:06.749351 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.749310    2579 scope.go:117] "RemoveContainer" containerID="5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2"
Sep 17 21:39:06.749705 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:39:06.749682    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2\": container with ID starting with 5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2 not found: ID does not exist" containerID="5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2"
Sep 17 21:39:06.749772 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.749712    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2"} err="failed to get container status \"5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2\": rpc error: code = NotFound desc = could not find container \"5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2\": container with ID starting with 5f97e1973ba2dabc95194af10c2b249762bf7a1371a283b8aa261dfa608fd9a2 not found: ID does not exist"
Sep 17 21:39:06.749772 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.749733    2579 scope.go:117] "RemoveContainer" containerID="169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd"
Sep 17 21:39:06.750027 ip-10-0-139-55 kubenswrapper[2579]: E0917 21:39:06.750003    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd\": container with ID starting with 169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd not found: ID does not exist" containerID="169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd"
Sep 17 21:39:06.750079 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.750034    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd"} err="failed to get container status \"169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd\": rpc error: code = NotFound desc = could not find container \"169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd\": container with ID starting with 169de06b3e4e983a9a1fc3b233462568183fe6d408e8342d92cae6d25c9223bd not found: ID does not exist"
Sep 17 21:39:06.751098 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.751081    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"]
Sep 17 21:39:06.754602 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:06.754583    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg"]
Sep 17 21:39:08.299068 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:08.299035    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" path="/var/lib/kubelet/pods/b1ebad23-a56e-4e98-9080-9d3743cbc953/volumes"
Sep 17 21:39:09.355942 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:09.355906    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:39:14.356591 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:14.356557    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:39:14.356990 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:14.356652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:39:19.238748 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:19.238700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:19.356452 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:19.356413    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:39:20.032082 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:20.032041    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:20.816034 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:20.816006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:21.677295 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:21.677266    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:22.477003 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:22.476899    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:39:22.493100 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:22.481813    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:39:22.493100 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:22.484176    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:39:22.493100 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:22.488998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:39:22.509109 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:22.509084    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:23.463293 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:23.463262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:24.255384 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:24.255352    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:24.355387 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:24.355354    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:39:25.048528 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:25.048497    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:25.821031 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:25.821006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:26.575578 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:26.575551    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:27.347378 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:27.347351    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:28.133007 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:28.132979    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:28.915056 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:28.915024    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:29.356041 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:29.356007    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:39:29.682513 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:29.682424    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:30.485496 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:30.485472    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3ccd0-6464d894c-mplj9_a6b61a42-da19-4b59-9933-16acdef775f0/switch-graph-3ccd0/0.log"
Sep 17 21:39:32.850178 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:32.850148    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6b61a42-da19-4b59-9933-16acdef775f0" containerID="2a807dea20fee82e89f39fd6d4a754880e36fcac537d35e8c2a4b5cf81fd9041" exitCode=137
Sep 17 21:39:32.850645 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:32.850227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" event={"ID":"a6b61a42-da19-4b59-9933-16acdef775f0","Type":"ContainerDied","Data":"2a807dea20fee82e89f39fd6d4a754880e36fcac537d35e8c2a4b5cf81fd9041"}
Sep 17 21:39:32.888596 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:32.888570    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:39:32.929084 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:32.929049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b61a42-da19-4b59-9933-16acdef775f0-openshift-service-ca-bundle\") pod \"a6b61a42-da19-4b59-9933-16acdef775f0\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") "
Sep 17 21:39:32.929237 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:32.929139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b61a42-da19-4b59-9933-16acdef775f0-proxy-tls\") pod \"a6b61a42-da19-4b59-9933-16acdef775f0\" (UID: \"a6b61a42-da19-4b59-9933-16acdef775f0\") "
Sep 17 21:39:32.929408 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:32.929385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b61a42-da19-4b59-9933-16acdef775f0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a6b61a42-da19-4b59-9933-16acdef775f0" (UID: "a6b61a42-da19-4b59-9933-16acdef775f0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:39:32.931283 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:32.931249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b61a42-da19-4b59-9933-16acdef775f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6b61a42-da19-4b59-9933-16acdef775f0" (UID: "a6b61a42-da19-4b59-9933-16acdef775f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:39:33.029708 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:33.029623    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b61a42-da19-4b59-9933-16acdef775f0-proxy-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:39:33.029708 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:33.029653    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b61a42-da19-4b59-9933-16acdef775f0-openshift-service-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 17 21:39:33.856233 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:33.856200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9" event={"ID":"a6b61a42-da19-4b59-9933-16acdef775f0","Type":"ContainerDied","Data":"b9bdb6770ae2a8cb69a05e547e7d0d4de24a5909f5560e656ae33c19b531b1ab"}
Sep 17 21:39:33.856233 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:33.856225    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"
Sep 17 21:39:33.856864 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:33.856256    2579 scope.go:117] "RemoveContainer" containerID="2a807dea20fee82e89f39fd6d4a754880e36fcac537d35e8c2a4b5cf81fd9041"
Sep 17 21:39:33.877583 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:33.877559    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"]
Sep 17 21:39:33.881708 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:33.881687    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9"]
Sep 17 21:39:34.298514 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:34.298440    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" path="/var/lib/kubelet/pods/a6b61a42-da19-4b59-9933-16acdef775f0/volumes"
Sep 17 21:39:38.283923 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:38.283893    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x66tf_1db2c98f-88ed-40a4-aa18-e2dc3a5bddd4/global-pull-secret-syncer/0.log"
Sep 17 21:39:38.425823 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:38.425793    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t6tpr_8dd712c9-6a04-4eeb-abe9-bc6f45dece63/konnectivity-agent/0.log"
Sep 17 21:39:38.472681 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:38.472651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-55.ec2.internal_3e878fac6c20a59b3277411748de820b/haproxy/0.log"
Sep 17 21:39:43.714415 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:43.714380    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vl9j4_0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e/node-exporter/0.log"
Sep 17 21:39:43.749148 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:43.749128    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vl9j4_0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e/kube-rbac-proxy/0.log"
Sep 17 21:39:43.792388 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:43.792366    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vl9j4_0f19e1b5-bcb0-4f59-bf0e-d4a00c5c596e/init-textfile/0.log"
Sep 17 21:39:44.328337 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.328306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-t9fln_5dae0b2e-b717-48f6-8042-81fd7c11054f/prometheus-operator/0.log"
Sep 17 21:39:44.397862 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.397840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-t9fln_5dae0b2e-b717-48f6-8042-81fd7c11054f/kube-rbac-proxy/0.log"
Sep 17 21:39:44.502171 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.502139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5684b66bd9-dbs92_0d5bdeea-8fe2-4fde-9fa3-85492620a4cf/telemeter-client/0.log"
Sep 17 21:39:44.531175 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.531146    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5684b66bd9-dbs92_0d5bdeea-8fe2-4fde-9fa3-85492620a4cf/reload/0.log"
Sep 17 21:39:44.554837 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.554811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5684b66bd9-dbs92_0d5bdeea-8fe2-4fde-9fa3-85492620a4cf/kube-rbac-proxy/0.log"
Sep 17 21:39:44.593906 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.593856    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869b875586-7s7pw_4c197e55-ac2a-495f-8ac4-328936334b88/thanos-query/0.log"
Sep 17 21:39:44.617423 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.617408    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869b875586-7s7pw_4c197e55-ac2a-495f-8ac4-328936334b88/kube-rbac-proxy-web/0.log"
Sep 17 21:39:44.641720 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.641699    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869b875586-7s7pw_4c197e55-ac2a-495f-8ac4-328936334b88/kube-rbac-proxy/0.log"
Sep 17 21:39:44.668645 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.668629    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869b875586-7s7pw_4c197e55-ac2a-495f-8ac4-328936334b88/prom-label-proxy/0.log"
Sep 17 21:39:44.697881 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.697861    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869b875586-7s7pw_4c197e55-ac2a-495f-8ac4-328936334b88/kube-rbac-proxy-rules/0.log"
Sep 17 21:39:44.725713 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:44.725694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869b875586-7s7pw_4c197e55-ac2a-495f-8ac4-328936334b88/kube-rbac-proxy-metrics/0.log"
Sep 17 21:39:45.811123 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811084    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"]
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811434    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container"
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811449    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container"
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811464    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kube-rbac-proxy"
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811470    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kube-rbac-proxy"
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811481    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kube-rbac-proxy"
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811486    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kube-rbac-proxy"
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811493    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2"
Sep 17 21:39:45.811500 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811499    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811512    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811518    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811527    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811535    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811587    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kserve-container"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811596    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ebad23-a56e-4e98-9080-9d3743cbc953" containerName="kube-rbac-proxy"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811604    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="695662f2-6e5c-40ed-a5af-9ccc683f51bb" containerName="splitter-graph-a1ff2"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811614    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kserve-container"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811622    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b61a42-da19-4b59-9933-16acdef775f0" containerName="switch-graph-3ccd0"
Sep 17 21:39:45.811755 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.811628    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ac1b73-0fa6-4b81-8750-cabf29254126" containerName="kube-rbac-proxy"
Sep 17 21:39:45.815684 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.815664    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:45.817922 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.817904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-44q44\"/\"kube-root-ca.crt\""
Sep 17 21:39:45.818692 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.818672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-44q44\"/\"openshift-service-ca.crt\""
Sep 17 21:39:45.818785 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.818673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-44q44\"/\"default-dockercfg-rvlpj\""
Sep 17 21:39:45.826614 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.826592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"]
Sep 17 21:39:45.931537 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.931508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-proc\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:45.931537 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.931538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqm5\" (UniqueName: \"kubernetes.io/projected/9337e57a-1649-4513-9e8c-2bee77f27dfa-kube-api-access-6wqm5\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:45.931731 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.931572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-lib-modules\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:45.931731 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.931637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-sys\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:45.931731 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:45.931679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-podres\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032354 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-podres\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032479 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-proc\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032479 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqm5\" (UniqueName: \"kubernetes.io/projected/9337e57a-1649-4513-9e8c-2bee77f27dfa-kube-api-access-6wqm5\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032479 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-lib-modules\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032479 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-proc\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032479 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-sys\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032709 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-podres\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032709 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-sys\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.032709 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.032594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9337e57a-1649-4513-9e8c-2bee77f27dfa-lib-modules\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.041708 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.041681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqm5\" (UniqueName: \"kubernetes.io/projected/9337e57a-1649-4513-9e8c-2bee77f27dfa-kube-api-access-6wqm5\") pod \"perf-node-gather-daemonset-6b242\" (UID: \"9337e57a-1649-4513-9e8c-2bee77f27dfa\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.124519 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.124494    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.274359 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.274303    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/1.log"
Sep 17 21:39:46.278909 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.278888    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-rp4v4_b7c8009c-f90e-4f56-943d-9e0b8e3414a8/console-operator/2.log"
Sep 17 21:39:46.392206 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.392180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"]
Sep 17 21:39:46.394599 ip-10-0-139-55 kubenswrapper[2579]: W0917 21:39:46.394568    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9337e57a_1649_4513_9e8c_2bee77f27dfa.slice/crio-3a7f4fef30f201e0d622ba3804b9031c1a7f7466c1aeba8739a49de47ad71ef4 WatchSource:0}: Error finding container 3a7f4fef30f201e0d622ba3804b9031c1a7f7466c1aeba8739a49de47ad71ef4: Status 404 returned error can't find the container with id 3a7f4fef30f201e0d622ba3804b9031c1a7f7466c1aeba8739a49de47ad71ef4
Sep 17 21:39:46.667847 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.667753    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56475bb9fb-8bgs9_6c57be80-41fd-42b1-9014-38e10c64b125/console/0.log"
Sep 17 21:39:46.699863 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.699838    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-h5nzh_32dd64bc-fb24-498d-a947-deb5e66032b9/download-server/0.log"
Sep 17 21:39:46.919082 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.918998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242" event={"ID":"9337e57a-1649-4513-9e8c-2bee77f27dfa","Type":"ContainerStarted","Data":"3bcc435c0723a09434eabffcd62af582ed2a3b22214ebe0fb59ba530f4ee6258"}
Sep 17 21:39:46.919082 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.919050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242" event={"ID":"9337e57a-1649-4513-9e8c-2bee77f27dfa","Type":"ContainerStarted","Data":"3a7f4fef30f201e0d622ba3804b9031c1a7f7466c1aeba8739a49de47ad71ef4"}
Sep 17 21:39:46.919082 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.919082    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:46.936180 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:46.936134    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242" podStartSLOduration=1.936119251 podStartE2EDuration="1.936119251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:39:46.934567367 +0000 UTC m=+3026.628376707" watchObservedRunningTime="2026-09-17 21:39:46.936119251 +0000 UTC m=+3026.629928587"
Sep 17 21:39:47.230100 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:47.230027    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-c9c2q_a7fb3dc3-f473-4fff-b4eb-28a2ce8a17ae/volume-data-source-validator/0.log"
Sep 17 21:39:48.329599 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:48.329571    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dtv9r_3a96b3a8-897d-4814-af96-b36564baba2d/dns/0.log"
Sep 17 21:39:48.353314 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:48.353291    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dtv9r_3a96b3a8-897d-4814-af96-b36564baba2d/kube-rbac-proxy/0.log"
Sep 17 21:39:48.450165 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:48.450139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xr4c8_b0f2f00d-88fd-4ecb-adb7-a1cc9836d870/dns-node-resolver/0.log"
Sep 17 21:39:48.874758 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:48.874731    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6445b7fc9f-gj4gh_adf57d26-bb0b-4a67-8208-8e375949907c/registry/0.log"
Sep 17 21:39:48.984042 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:48.984018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cspdc_2193e970-4de6-43f0-ab19-e94574c01546/node-ca/0.log"
Sep 17 21:39:50.068926 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:50.068896    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8hwnb_030b8144-24d5-47f2-a965-5c9fe63f7882/serve-healthcheck-canary/0.log"
Sep 17 21:39:50.627001 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:50.626973    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wbwfl_c680a9a3-ba38-4887-9ece-1eaca21028e2/kube-rbac-proxy/0.log"
Sep 17 21:39:50.645309 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:50.645276    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wbwfl_c680a9a3-ba38-4887-9ece-1eaca21028e2/exporter/0.log"
Sep 17 21:39:50.665017 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:50.664976    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wbwfl_c680a9a3-ba38-4887-9ece-1eaca21028e2/extractor/0.log"
Sep 17 21:39:52.933928 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:52.933898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-6b242"
Sep 17 21:39:52.976969 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:52.976940    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-66f4b8957f-b4kdd_3b46c6e2-cec1-412f-81ef-30252529a6bf/manager/0.log"
Sep 17 21:39:53.032762 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:53.032736    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-fnlkh_b9b96f7b-158c-473a-ae3a-4b46471f62d2/server/0.log"
Sep 17 21:39:58.946884 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:58.946858    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-76468_3d96d7a2-ce04-4878-9f9f-e8f3ec947d56/kube-multus/0.log"
Sep 17 21:39:59.136217 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.136184    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4sftt_720792d8-c7ab-4c51-8ce1-7a79a051e9af/kube-multus-additional-cni-plugins/0.log"
Sep 17 21:39:59.156662 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.156640    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4sftt_720792d8-c7ab-4c51-8ce1-7a79a051e9af/egress-router-binary-copy/0.log"
Sep 17 21:39:59.176788 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.176769    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4sftt_720792d8-c7ab-4c51-8ce1-7a79a051e9af/cni-plugins/0.log"
Sep 17 21:39:59.197499 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.197454    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4sftt_720792d8-c7ab-4c51-8ce1-7a79a051e9af/bond-cni-plugin/0.log"
Sep 17 21:39:59.218183 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.218161    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4sftt_720792d8-c7ab-4c51-8ce1-7a79a051e9af/routeoverride-cni/0.log"
Sep 17 21:39:59.238063 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.238046    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4sftt_720792d8-c7ab-4c51-8ce1-7a79a051e9af/whereabouts-cni-bincopy/0.log"
Sep 17 21:39:59.259865 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.259848    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4sftt_720792d8-c7ab-4c51-8ce1-7a79a051e9af/whereabouts-cni/0.log"
Sep 17 21:39:59.577415 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.577388    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sn8wn_410d7742-1dd0-4ca9-abcf-3b999c572cbf/network-metrics-daemon/0.log"
Sep 17 21:39:59.598004 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:39:59.597979    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sn8wn_410d7742-1dd0-4ca9-abcf-3b999c572cbf/kube-rbac-proxy/0.log"
Sep 17 21:40:00.352584 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.352557    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-controller/0.log"
Sep 17 21:40:00.369335 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.369297    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/0.log"
Sep 17 21:40:00.369717 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.369686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovn-acl-logging/1.log"
Sep 17 21:40:00.388307 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.388281    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/kube-rbac-proxy-node/0.log"
Sep 17 21:40:00.408343 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.408309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 21:40:00.430801 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.430783    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/northd/0.log"
Sep 17 21:40:00.451333 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.451295    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/nbdb/0.log"
Sep 17 21:40:00.475108 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.475088    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/sbdb/0.log"
Sep 17 21:40:00.512924 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:00.512902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khg2b_336e56a1-cbd3-497b-b784-4890ae07845a/ovnkube-controller/0.log"
Sep 17 21:40:01.950107 ip-10-0-139-55 kubenswrapper[2579]: I0917 21:40:01.950079    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-r5xpp_3df1144b-3aa8-4688-aa04-2e16ff789302/network-check-target-container/0.log"