Sep 09 20:11:22.857782 ip-10-0-137-234 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:11:23.213116 ip-10-0-137-234 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:23.213116 ip-10-0-137-234 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:11:23.213116 ip-10-0-137-234 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:23.213116 ip-10-0-137-234 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:11:23.213116 ip-10-0-137-234 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:23.213909 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.213821    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:11:23.218247 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218231    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:23.218247 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218247    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218251    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218254    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218257    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218260    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218264    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218266    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218269    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218272    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218275    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218278    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218281    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218283    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218286    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218289    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218292    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218294    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218297    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218300    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218303    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:23.218309 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218306    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218308    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218311    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218314    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218317    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218320    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218323    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218326    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218328    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218331    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218334    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218336    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218339    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218341    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218344    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218346    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218350    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218352    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218355    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:23.218809 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218358    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218360    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218363    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218365    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218368    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218370    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218373    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218376    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218378    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218381    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218383    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218386    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218388    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218391    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218393    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218396    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218399    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218402    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218406    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218410    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:23.219279 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218413    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218416    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218419    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218422    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218426    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218429    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218432    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218435    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218439    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218441    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218444    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218447    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218449    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218452    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218456    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218459    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218462    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218465    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218468    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:23.219775 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218470    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218473    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218476    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218478    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218481    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218483    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218486    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218488    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218504    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218507    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218510    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218512    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218515    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218518    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218520    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218523    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218526    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218529    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218532    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.218534    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:23.220239 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219205    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219212    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219214    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219218    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219220    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219223    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219226    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219228    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219231    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219235    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219237    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219240    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219243    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219245    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219248    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219251    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219254    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219256    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219259    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219262    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:23.220741 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219266    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219268    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219271    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219274    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219277    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219279    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219282    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219285    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219287    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219290    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219293    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219296    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219299    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219301    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219305    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219309    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219312    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219315    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219318    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:23.221240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219320    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219323    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219326    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219329    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219332    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219335    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219337    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219340    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219343    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219346    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219349    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219351    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219354    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219357    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219360    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219362    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219365    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219367    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219370    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219372    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:23.221819 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219375    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219377    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219380    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219384    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219388    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219390    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219393    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219395    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219398    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219401    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219403    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219406    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219408    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219411    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219413    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219417    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219421    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219425    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219428    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:23.222316 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219431    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219434    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219438    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219441    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219443    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219446    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219449    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219452    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219454    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219457    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219460    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219462    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219465    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219467    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219470    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219473    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219475    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219478    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219481    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219483    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:23.222796 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219486    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219894    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219900    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219904    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219908    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219912    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219916    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219919    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219922    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219926    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219929    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219932    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219935    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219938    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219941    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219944    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219946    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219949    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219952    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:23.223303 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219955    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219958    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219960    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219963    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219966    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219969    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219971    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219974    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219976    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219979    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219981    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219984    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219987    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219989    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219992    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219996    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.219998    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220001    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220004    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220006    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:23.223781 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220009    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220011    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220014    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220017    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220019    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220022    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220025    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220027    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220029    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220032    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220037    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220040    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220042    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220045    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220048    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220050    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220053    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220055    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220058    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220061    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:23.224268 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220063    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220066    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220068    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220071    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220074    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220076    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220079    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220081    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220084    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220086    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220089    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220092    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220094    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220097    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220100    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220102    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220105    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220108    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220110    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220113    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:23.224802 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220116    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220119    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220122    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220125    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220127    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220130    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220133    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220137    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220139    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220141    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220144    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220147    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220149    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220152    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220155    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220157    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220160    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220162    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220165    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:23.225284 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220167    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220170    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220249    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220282    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220288    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220292    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220296    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220300    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220304    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220309    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220313    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220316    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220320    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220323    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220326    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220329    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220333    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220336    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220339    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220342    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220349    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220351    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220354    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220358    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:11:23.225766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220361    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220364    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220367    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220370    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220374    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220377    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220380    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220383    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220386    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220389    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220392    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220395    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220398    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220401    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220404    2562 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220407    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220411    2562 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220413    2562 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220416    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220420    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220423    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220427    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220429    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220432    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220436    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:11:23.226381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220439    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220442    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220445    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220447    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220450    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220453    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220456    2562 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220460    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220463    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220466    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220469    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220472    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220475    2562 flags.go:64] FLAG: --help="false"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220478    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220481    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220484    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220487    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220503    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220507    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220510    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220513    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220516    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220519    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:11:23.226994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220522    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220525    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220528    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220531    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220535    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220538    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220541    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220544    2562 flags.go:64] FLAG: --lock-file=""
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220547    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220550    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220553    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220559    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220562    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220564    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220567    2562 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220571    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220574    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220577    2562 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220580    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220584    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220587    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220591    2562 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220594    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220598    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220601    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:11:23.227584 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220604    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220607    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220610    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220613    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220620    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220623    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220626    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220643    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220647    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220653    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220657    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220660    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220664    2562 flags.go:64] FLAG: --port="10250"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220667    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220670    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0730d25b1e172d64c"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220673    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220676    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220679    2562 flags.go:64] FLAG: --register-node="true"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220682    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220686    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220689    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220692    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220694    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220698    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220701    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:11:23.228182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220704    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220707    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220710    2562 flags.go:64] FLAG: --runonce="false"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220713    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220716    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220719    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220722    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220725    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220728    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220731    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220734    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220737    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220740    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220743    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220746    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220749    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220752    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220756    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220761    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220764    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220768    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220774    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220778    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220781    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220784    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:11:23.228820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220788    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220791    2562 flags.go:64] FLAG: --v="2"
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220794    2562 flags.go:64] FLAG: --version="false"
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220798    2562 flags.go:64] FLAG: --vmodule=""
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220802    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.220805    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220916    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220920    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220923    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220926    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220929    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220932    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220935    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220937    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220940    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220943    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220945    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220948    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220951    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220953    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220956    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:23.229415 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220959    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220961    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220964    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220966    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220969    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220971    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220974    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220984    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220988    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220992    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220995    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.220998    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221001    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221004    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221007    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221009    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221014    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221017    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221020    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:23.229973 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221022    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221025    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221028    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221030    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221033    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221035    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221038    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221040    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221043    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221046    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221048    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221051    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221053    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221056    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221058    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221061    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221063    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221066    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221069    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:23.230433 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221071    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221076    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221088    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221091    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221094    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221097    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221099    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221102    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221105    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221107    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221110    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221112    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221115    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221117    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221120    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221123    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221125    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221128    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221131    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:23.230922 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221133    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221136    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221138    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221141    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221145    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221149    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221153    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221157    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221160    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221162    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221165    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221168    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221170    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221173    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221177    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221180    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221189    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221192    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221194    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221197    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:23.231404 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221200    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221202    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221205    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221208    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221210    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221213    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.221215    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.221993    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.228799    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.228813    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.228833    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.228840    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228895    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228899    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228903    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:23.231929 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228906    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228909    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228912    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228915    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228918    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228920    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228923    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228926    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228929    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228931    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228934    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228938    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228940    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228943    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228946    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228948    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228951    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228953    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228956    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228959    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:23.232322 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228961    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228964    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228966    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228969    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228972    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228974    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228978    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228981    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228985    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228987    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228990    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228992    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228995    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.228997    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229000    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229003    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229005    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229008    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229010    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229013    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:23.232816 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229015    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229018    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229021    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229023    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229026    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229029    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229031    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229034    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229036    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229039    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229041    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229044    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229047    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229049    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229052    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229055    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229057    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229060    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229063    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229070    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:23.233341 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229072    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229075    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229078    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229081    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229083    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229088    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229091    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229094    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229097    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229100    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229103    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229106    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229109    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229111    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229114    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229116    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229119    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229121    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229124    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:23.233862 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229128    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229131    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229134    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229137    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229140    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229143    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229145    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229148    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229151    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229154    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229156    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229160    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229163    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229167    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229171    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229174    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:23.234333 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229177    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.229182    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229294    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229300    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229304    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229307    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229310    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229313    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229316    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229319    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229322    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229325    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229328    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229331    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:23.234750 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229334    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229337    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229340    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229342    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229345    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229348    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229350    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229353    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229355    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229358    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229360    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229363    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229367    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229370    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229373    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229376    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229379    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229382    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229384    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229387    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:23.235097 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229390    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229392    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229395    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229397    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229400    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229402    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229405    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229407    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229410    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229412    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229415    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229418    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229421    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229424    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229426    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229429    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229432    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229434    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229437    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:23.235615 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229439    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229442    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229444    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229447    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229449    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229452    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229454    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229457    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229460    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229462    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229465    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229468    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229471    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229473    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229476    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229478    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229481    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229483    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229486    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229489    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229509    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:23.236069 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229513    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229515    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229518    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229521    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229524    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229526    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229529    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229531    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229534    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229538    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229541    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229544    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229547    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229550    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229552    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229555    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229558    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229561    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229564    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:23.236686 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229567    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229570    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229572    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229575    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229578    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229581    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229583    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:23.229586    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.229591    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.229696    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.231931    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.232797    2562 server.go:1019] "Starting client certificate rotation"
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.232894    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:23.237143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.232938    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:23.237458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.235133    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:23.238866 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.238846    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:23.250095 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.250076    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:11:23.253756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.253737    2562 log.go:25] "Validated CRI v1 image API"
Sep 09 20:11:23.254629 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.254605    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:11:23.257468 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.257448    2562 fs.go:135] Filesystem UUIDs: map[20b61623-c8d3-44cd-918f-ffbee0f0fd85:/dev/nvme0n1p3 585cbea0-fca2-418e-91ad-78ce72d3d431:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 20:11:23.257551 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.257466    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:11:23.262278 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.262164    2562 manager.go:216] Machine: {Timestamp:2026-09-09 20:11:23.261108567 +0000 UTC m=+0.301385734 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111096 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc590a3874ee26324b4f386ba97c0 SystemUUID:ec2dc590-a387-4ee2-6324-b4f386ba97c0 BootID:39625bd8-97c2-4c90-9345-5b37c121d966 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7a:87:6c:38:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7a:87:6c:38:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:74:d7:ed:78:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:11:23.262278 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.262274    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:11:23.262382 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.262350    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:11:23.263784 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.263769    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:23.265022 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.264992    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:11:23.265157 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.265024    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:11:23.265760 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.265748    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:11:23.265799 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.265763    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:11:23.265799 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.265770    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:23.267380 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.267369    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:23.268851 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.268842    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:11:23.272778 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.272767    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:23.272891 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.272882    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:11:23.272925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.272912    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:11:23.272925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.272921    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:11:23.272975 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.272932    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:11:23.272975 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.272941    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:11:23.272975 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.272951    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:11:23.273324 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.273313    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:23.273363 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.273331    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:23.275350 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.275332    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:11:23.277622 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.277603    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:11:23.277703 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.277644    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278361    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278382    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278391    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278399    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278410    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278419    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278438    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278446    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278454    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278517    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278546    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:11:23.278692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.278554    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:11:23.279185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.279172    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:11:23.279239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.279187    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:11:23.282868 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.282839    2562 server.go:1268] "Started kubelet"
Sep 09 20:11:23.283025 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.282957    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:11:23.283086 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.283023    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:11:23.283086 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.283048    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:11:23.283788 ip-10-0-137-234 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:11:23.285036 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.285004    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:11:23.287709 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.287679    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:11:23.287779 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.287713    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7wtm2"
Sep 09 20:11:23.287894 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.287877    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:11:23.287993 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.287876    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:11:23.289541 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.289519    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:11:23.290767 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.290723    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:11:23.292735 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.292715    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:23.293221 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293208    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:11:23.293390 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.293364    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.293800 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293779    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7wtm2"
Sep 09 20:11:23.293800 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293783    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:11:23.293936 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293785    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:11:23.293936 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293818    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:11:23.293936 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293915    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:11:23.293936 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293923    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:11:23.294126 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293943    2562 factory.go:55] Registering systemd factory
Sep 09 20:11:23.294126 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.293957    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:11:23.294218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.294171    2562 factory.go:153] Registering CRI-O factory
Sep 09 20:11:23.294218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.294186    2562 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:11:23.294286 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.294237    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:11:23.294286 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.294270    2562 factory.go:103] Registering Raw factory
Sep 09 20:11:23.294286 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.294283    2562 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:11:23.294717 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.294702    2562 manager.go:349] Starting recovery of all containers
Sep 09 20:11:23.295366 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.294294    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-234.ec2.internal.18d3c0174bafba29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-234.ec2.internal,UID:ip-10-0-137-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-234.ec2.internal,},FirstTimestamp:2026-09-09 20:11:23.282807337 +0000 UTC m=+0.323084504,LastTimestamp:2026-09-09 20:11:23.282807337 +0000 UTC m=+0.323084504,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-234.ec2.internal,}"
Sep 09 20:11:23.304538 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.304329    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:23.306403 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.306387    2562 manager.go:354] Recovery completed
Sep 09 20:11:23.307256 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.307236    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-234.ec2.internal\" not found" node="ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.311369 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.311140    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:23.313014 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.313000    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:23.313068 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.313026    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:23.313068 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.313044    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:23.313566 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.313555    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:11:23.313566 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.313566    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:11:23.313654 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.313590    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:23.313654 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.313622    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:23.315659 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.315644    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:23.315659 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.315654    2562 policy_none.go:49] "None policy: Start"
Sep 09 20:11:23.315791 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.315671    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:11:23.315791 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.315688    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:11:23.315791 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.315692    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:23.315791 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.315717    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:23.317717 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.317701    2562 policy_none.go:47] "Start"
Sep 09 20:11:23.344054 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.344039    2562 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.344072    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.344083    2562 server.go:87] "Starting device plugin registration server"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.344364    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.344376    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.344477    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.344580    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.344591    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.345087    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:11:23.367466 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.345128    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.391725 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.391697    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:11:23.392857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.392839    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:11:23.392926 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.392887    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:11:23.392926 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.392911    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:11:23.393022 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.392956    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:11:23.395253 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.395235    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:23.445140 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.445115    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:23.446068 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.446047    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:23.446185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.446080    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:23.446185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.446093    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:23.446185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.446116    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.453646 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.453631    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.453689 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.453654    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-234.ec2.internal\": node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.475198 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.475180    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.494072 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.494043    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"]
Sep 09 20:11:23.494147 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.494133    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:23.495079 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.495062    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:23.495164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.495093    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:23.495164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.495106    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:23.497225 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497213    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:23.497342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497325    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.497385 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497364    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:23.497957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497932    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:23.497957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497957    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:23.498087 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497934    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:23.498087 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497988    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:23.498087 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.498003    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:23.498087 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.497967    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:23.500120 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.500104    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.500198 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.500131    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:23.500885 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.500871    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:23.500957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.500894    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:23.500957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.500904    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:23.521110 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.521094    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-234.ec2.internal\" not found" node="ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.525353 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.525337    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-234.ec2.internal\" not found" node="ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.575917 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.575901    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.676640 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.676615    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.694944 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.694921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f674f484fa389b34823ea538de3d17f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-234.ec2.internal\" (UID: \"2f674f484fa389b34823ea538de3d17f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.695002 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.694948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6277271f02f9fd3891a478b286fe5b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"c6277271f02f9fd3891a478b286fe5b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.695002 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.694966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6277271f02f9fd3891a478b286fe5b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"c6277271f02f9fd3891a478b286fe5b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.777385 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.777314    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.795579 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.795556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f674f484fa389b34823ea538de3d17f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-234.ec2.internal\" (UID: \"2f674f484fa389b34823ea538de3d17f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.795633 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.795582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6277271f02f9fd3891a478b286fe5b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"c6277271f02f9fd3891a478b286fe5b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.795633 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.795603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6277271f02f9fd3891a478b286fe5b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"c6277271f02f9fd3891a478b286fe5b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.795694 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.795663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f674f484fa389b34823ea538de3d17f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-234.ec2.internal\" (UID: \"2f674f484fa389b34823ea538de3d17f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.795694 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.795670    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6277271f02f9fd3891a478b286fe5b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"c6277271f02f9fd3891a478b286fe5b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.795756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.795678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6277271f02f9fd3891a478b286fe5b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"c6277271f02f9fd3891a478b286fe5b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.823686 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.823666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.828575 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:23.828557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:23.878309 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.878286    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:23.978821 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:23.978780    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:24.079321 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:24.079260    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:25.246332 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:25.246307    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:25.298733 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.298697    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:11:25.299158 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.298863    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:25.299158 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.298880    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:25.347168 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:25.347141    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:25.364023 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.363980    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:06:24 +0000 UTC" deadline="2028-05-11 17:38:43.414153568 +0000 UTC"
Sep 09 20:11:25.364023 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.364023    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14637h27m18.050135922s"
Sep 09 20:11:25.447413 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:25.447354    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Sep 09 20:11:25.477266 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.477240    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:25.494155 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.494138    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:25.494566 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:25.494542    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f674f484fa389b34823ea538de3d17f.slice/crio-06b069021f7cc6fffec665faf7ab3d1da0ddbde6a3aba7d52f80053db94a94c4 WatchSource:0}: Error finding container 06b069021f7cc6fffec665faf7ab3d1da0ddbde6a3aba7d52f80053db94a94c4: Status 404 returned error can't find the container with id 06b069021f7cc6fffec665faf7ab3d1da0ddbde6a3aba7d52f80053db94a94c4
Sep 09 20:11:25.494805 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:25.494784    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6277271f02f9fd3891a478b286fe5b5.slice/crio-cbaf542e7a561cdd18e9dd1b298c46ef549385588500b912207ca8e00b6d9ca2 WatchSource:0}: Error finding container cbaf542e7a561cdd18e9dd1b298c46ef549385588500b912207ca8e00b6d9ca2: Status 404 returned error can't find the container with id cbaf542e7a561cdd18e9dd1b298c46ef549385588500b912207ca8e00b6d9ca2
Sep 09 20:11:25.499160 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.499140    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:11:25.560358 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.560337    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:25.571504 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.571487    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:25.572352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.572340    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Sep 09 20:11:25.580919 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:25.580907    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:26.257787 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.257755    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:26.340592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.340558    2562 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:26.349919 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.349892    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:26.350340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.350312    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-zq8t9","kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal","openshift-multus/multus-k5bpz","openshift-network-diagnostics/network-check-target-sdx6r","openshift-network-operator/iptables-alerter-g24n5","openshift-ovn-kubernetes/ovnkube-node-xt4pz","kube-system/global-pull-secret-syncer-qkfk2","kube-system/konnectivity-agent-lkdsq","openshift-cluster-node-tuning-operator/tuned-7dzkh","openshift-dns/node-resolver-2cdv6","openshift-image-registry/node-ca-n6pfl","openshift-multus/multus-additional-cni-plugins-bm5wh"]
Sep 09 20:11:26.352935 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.352893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.355079 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.355056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.356021 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.355970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rr7s2\""
Sep 09 20:11:26.356114 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.356085    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:26.356114 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.356107    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:26.357233 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.357214    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gtwl\""
Sep 09 20:11:26.357954 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.357469    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:26.357954 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.357550    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:26.357954 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.357479    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.357954 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.357469    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:26.359477 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.359454    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:26.359605 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.359567    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.360197 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.360175    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:26.360292 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.360205    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:26.360292 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.360176    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n6qqv\""
Sep 09 20:11:26.360292 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.360175    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:26.361815 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.361664    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:26.362014 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.361987    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.362513 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.362490    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hwxjp\""
Sep 09 20:11:26.362609 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.362597    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:26.363550 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.362757    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:26.363550 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.362815    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:26.363550 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.362860    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:26.364220 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.364200    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:26.364308 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.364275    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:26.364782 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.364200    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:26.364782 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.364629    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4shnv\""
Sep 09 20:11:26.366389 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.366361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.366503 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.366443    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:26.368572 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.368554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0762481-ab10-46dd-a065-727d330ce5e0-cni-binary-copy\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/512ceb34-1d30-47d0-b1f4-e2a0871e519b-serviceca\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvrx\" (UniqueName: \"kubernetes.io/projected/512ceb34-1d30-47d0-b1f4-e2a0871e519b-kube-api-access-tvvrx\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-hostroot\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/512ceb34-1d30-47d0-b1f4-e2a0871e519b-host\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-device-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.369747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cni-binary-copy\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-cni-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-socket-dir-parent\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369842    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-k8s-cni-cncf-io\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-netns\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b0762481-ab10-46dd-a065-727d330ce5e0-multus-daemon-config\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-socket-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdf7p\" (UniqueName: \"kubernetes.io/projected/040eebcb-98c6-4c25-99f3-c9a0f400a644-kube-api-access-fdf7p\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.369978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-system-cni-dir\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-cni-bin\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvvf\" (UniqueName: \"kubernetes.io/projected/b0762481-ab10-46dd-a065-727d330ce5e0-kube-api-access-lcvvf\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370081    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-conf-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-etc-kubernetes\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.370231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-system-cni-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-registration-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-cni-multus\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-kubelet\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1433c535-6ce0-4f9b-b94c-a42e534ff3ee-agent-certs\") pod \"konnectivity-agent-lkdsq\" (UID: \"1433c535-6ce0-4f9b-b94c-a42e534ff3ee\") " pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1433c535-6ce0-4f9b-b94c-a42e534ff3ee-konnectivity-ca\") pod \"konnectivity-agent-lkdsq\" (UID: \"1433c535-6ce0-4f9b-b94c-a42e534ff3ee\") " pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qpm\" (UniqueName: \"kubernetes.io/projected/c038dca1-e9c9-4c1a-aefb-4125c06f7371-kube-api-access-w9qpm\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-cnibin\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cnibin\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-multus-certs\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.370705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-sys-fs\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.371112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.371117    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.372330 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.372303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lmb6g\""
Sep 09 20:11:26.372421 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.372361    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:26.372729 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.372671    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:26.372729 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.372710    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:26.373080 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.373062    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:26.373871 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.373546    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:26.373871 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.373568    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:26.374311 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.374281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:26.374425 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.374390    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:26.374590 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.374570    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:26.376032 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.375996    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:26.376372 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.376350    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:26.376458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.376432    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pl5fx\""
Sep 09 20:11:26.376516 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.376464    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:26.378576 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.378551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.381058 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.380996    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:26.381058 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.381040    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:26.381237 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.381066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.381392 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.381320    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bz4t4\""
Sep 09 20:11:26.383203 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.383182    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5jvv2\""
Sep 09 20:11:26.383287 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.383205    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:26.383287 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.383282    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:26.393797 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.393776    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7q5wp"
Sep 09 20:11:26.399954 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.399933    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7q5wp"
Sep 09 20:11:26.462551 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.462527    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:26.464632 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.464590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" event={"ID":"c6277271f02f9fd3891a478b286fe5b5","Type":"ContainerStarted","Data":"cbaf542e7a561cdd18e9dd1b298c46ef549385588500b912207ca8e00b6d9ca2"}
Sep 09 20:11:26.466168 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.466145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal" event={"ID":"2f674f484fa389b34823ea538de3d17f","Type":"ContainerStarted","Data":"06b069021f7cc6fffec665faf7ab3d1da0ddbde6a3aba7d52f80053db94a94c4"}
Sep 09 20:11:26.471445 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-host\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.471541 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471459    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-cni-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.471541 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvvf\" (UniqueName: \"kubernetes.io/projected/b0762481-ab10-46dd-a065-727d330ce5e0-kube-api-access-lcvvf\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.471653 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-cni-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.471653 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.471653 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-socket-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdf7p\" (UniqueName: \"kubernetes.io/projected/040eebcb-98c6-4c25-99f3-c9a0f400a644-kube-api-access-fdf7p\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-cni-bin\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-var-lib-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-sys\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-conf-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471846    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-cni-bin\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:26.471872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-socket-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-cni-bin\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-system-cni-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-conf-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.471996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-system-cni-dir\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-registration-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-cni-multus\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-registration-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-kubelet\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-cni-multus\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysconfig\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1433c535-6ce0-4f9b-b94c-a42e534ff3ee-agent-certs\") pod \"konnectivity-agent-lkdsq\" (UID: \"1433c535-6ce0-4f9b-b94c-a42e534ff3ee\") " pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.472325 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-var-lib-kubelet\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a80d1370-4cbe-4f65-8dd5-d42457f347b2-kubelet-config\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-host-slash\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472371    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-env-overrides\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovn-node-metrics-cert\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-systemd\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-var-lib-kubelet\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8335a8ba-a535-4f1b-b63d-285c7107f06c-tmp\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-multus-certs\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472544    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqm9\" (UniqueName: \"kubernetes.io/projected/8335a8ba-a535-4f1b-b63d-285c7107f06c-kube-api-access-tcqm9\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-sys-fs\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a80d1370-4cbe-4f65-8dd5-d42457f347b2-dbus\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-run\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-log-socket\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvrx\" (UniqueName: \"kubernetes.io/projected/512ceb34-1d30-47d0-b1f4-e2a0871e519b-kube-api-access-tvvrx\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-hostroot\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.473213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-kubelet\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-systemd\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-cni-netd\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/512ceb34-1d30-47d0-b1f4-e2a0871e519b-host\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-device-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cni-binary-copy\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-socket-dir-parent\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-k8s-cni-cncf-io\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472966    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.472989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-netns\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b0762481-ab10-46dd-a065-727d330ce5e0-multus-daemon-config\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-netns\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-ovn\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-system-cni-dir\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473300    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-hostroot\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-device-dir\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-multus-certs\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/040eebcb-98c6-4c25-99f3-c9a0f400a644-sys-fs\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/512ceb34-1d30-47d0-b1f4-e2a0871e519b-host\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-system-cni-dir\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-multus-socket-dir-parent\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-systemd-units\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-host-run-k8s-cni-cncf-io\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-slash\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovnkube-script-lib\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qng8c\" (UniqueName: \"kubernetes.io/projected/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-kube-api-access-qng8c\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-tuned\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-etc-kubernetes\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllqt\" (UniqueName: \"kubernetes.io/projected/b355c14c-96e4-4d5d-ada5-6142cd99e951-kube-api-access-vllqt\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-kubernetes\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473636    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysctl-d\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysctl-conf\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-etc-kubernetes\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.474857 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-etc-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-tmp-dir\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1433c535-6ce0-4f9b-b94c-a42e534ff3ee-konnectivity-ca\") pod \"konnectivity-agent-lkdsq\" (UID: \"1433c535-6ce0-4f9b-b94c-a42e534ff3ee\") " pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qpm\" (UniqueName: \"kubernetes.io/projected/c038dca1-e9c9-4c1a-aefb-4125c06f7371-kube-api-access-w9qpm\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b0762481-ab10-46dd-a065-727d330ce5e0-multus-daemon-config\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.473971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cni-binary-copy\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-cnibin\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-iptables-alerter-script\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-node-log\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474125    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0762481-ab10-46dd-a065-727d330ce5e0-cnibin\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cnibin\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovnkube-config\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cnibin\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-hosts-file\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.475756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lslt\" (UniqueName: \"kubernetes.io/projected/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-kube-api-access-4lslt\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0762481-ab10-46dd-a065-727d330ce5e0-cni-binary-copy\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1433c535-6ce0-4f9b-b94c-a42e534ff3ee-konnectivity-ca\") pod \"konnectivity-agent-lkdsq\" (UID: \"1433c535-6ce0-4f9b-b94c-a42e534ff3ee\") " pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl75r\" (UniqueName: \"kubernetes.io/projected/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-kube-api-access-fl75r\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-run-netns\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/512ceb34-1d30-47d0-b1f4-e2a0871e519b-serviceca\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-modprobe-d\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474463    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-lib-modules\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.474907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/512ceb34-1d30-47d0-b1f4-e2a0871e519b-serviceca\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.475077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c038dca1-e9c9-4c1a-aefb-4125c06f7371-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.476592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.475617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0762481-ab10-46dd-a065-727d330ce5e0-cni-binary-copy\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.477705 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.477675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1433c535-6ce0-4f9b-b94c-a42e534ff3ee-agent-certs\") pod \"konnectivity-agent-lkdsq\" (UID: \"1433c535-6ce0-4f9b-b94c-a42e534ff3ee\") " pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.481187 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.481143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvvf\" (UniqueName: \"kubernetes.io/projected/b0762481-ab10-46dd-a065-727d330ce5e0-kube-api-access-lcvvf\") pod \"multus-k5bpz\" (UID: \"b0762481-ab10-46dd-a065-727d330ce5e0\") " pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.484024 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.483458    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qpm\" (UniqueName: \"kubernetes.io/projected/c038dca1-e9c9-4c1a-aefb-4125c06f7371-kube-api-access-w9qpm\") pod \"multus-additional-cni-plugins-bm5wh\" (UID: \"c038dca1-e9c9-4c1a-aefb-4125c06f7371\") " pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.485120 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.485076    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdf7p\" (UniqueName: \"kubernetes.io/projected/040eebcb-98c6-4c25-99f3-c9a0f400a644-kube-api-access-fdf7p\") pod \"aws-ebs-csi-driver-node-6fmz8\" (UID: \"040eebcb-98c6-4c25-99f3-c9a0f400a644\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.485379 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.485356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvrx\" (UniqueName: \"kubernetes.io/projected/512ceb34-1d30-47d0-b1f4-e2a0871e519b-kube-api-access-tvvrx\") pod \"node-ca-n6pfl\" (UID: \"512ceb34-1d30-47d0-b1f4-e2a0871e519b\") " pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.574873 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a80d1370-4cbe-4f65-8dd5-d42457f347b2-dbus\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.574873 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-run\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.574873 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-log-socket\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-kubelet\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-systemd\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-run\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a80d1370-4cbe-4f65-8dd5-d42457f347b2-dbus\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-log-socket\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-kubelet\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.574984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-cni-netd\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-ovn\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-systemd\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.575040    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-systemd-units\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-ovn\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-slash\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-cni-netd\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-run-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-systemd-units\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.575136    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret podName:a80d1370-4cbe-4f65-8dd5-d42457f347b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:27.075100637 +0000 UTC m=+3.048870699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret") pod "global-pull-secret-syncer-qkfk2" (UID: "a80d1370-4cbe-4f65-8dd5-d42457f347b2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovnkube-script-lib\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-slash\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qng8c\" (UniqueName: \"kubernetes.io/projected/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-kube-api-access-qng8c\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-tuned\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllqt\" (UniqueName: \"kubernetes.io/projected/b355c14c-96e4-4d5d-ada5-6142cd99e951-kube-api-access-vllqt\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-kubernetes\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysctl-d\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysctl-conf\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-etc-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-tmp-dir\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-etc-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysctl-d\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-iptables-alerter-script\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575474    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysctl-conf\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.575925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-node-log\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-kubernetes\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovnkube-config\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-hosts-file\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575563    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lslt\" (UniqueName: \"kubernetes.io/projected/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-kube-api-access-4lslt\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-node-log\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl75r\" (UniqueName: \"kubernetes.io/projected/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-kube-api-access-fl75r\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-tmp-dir\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-run-netns\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-modprobe-d\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-lib-modules\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-host\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-var-lib-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovnkube-script-lib\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-sys\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.576696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-sys\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-lib-modules\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-host\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-run-netns\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-hosts-file\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-var-lib-openvswitch\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-cni-bin\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-modprobe-d\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.575978    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysconfig\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a80d1370-4cbe-4f65-8dd5-d42457f347b2-kubelet-config\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovnkube-config\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-iptables-alerter-script\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.576069    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-host-slash\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.577501 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-host-slash\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a80d1370-4cbe-4f65-8dd5-d42457f347b2-kubelet-config\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576024    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-host-cni-bin\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576062    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-sysconfig\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.576119    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:27.076101907 +0000 UTC m=+3.049871965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-env-overrides\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovn-node-metrics-cert\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-systemd\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-var-lib-kubelet\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8335a8ba-a535-4f1b-b63d-285c7107f06c-tmp\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqm9\" (UniqueName: \"kubernetes.io/projected/8335a8ba-a535-4f1b-b63d-285c7107f06c-kube-api-access-tcqm9\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-systemd\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-env-overrides\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.576533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8335a8ba-a535-4f1b-b63d-285c7107f06c-var-lib-kubelet\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.578080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8335a8ba-a535-4f1b-b63d-285c7107f06c-etc-tuned\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.578908 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.578741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-ovn-node-metrics-cert\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.578908 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.578837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8335a8ba-a535-4f1b-b63d-285c7107f06c-tmp\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.582483 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.582458    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:26.582483 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.582485    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:26.582650 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.582500    2562 projected.go:196] Error preparing data for projected volume kube-api-access-qh76s for pod openshift-network-diagnostics/network-check-target-sdx6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:26.582650 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:26.582566    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s podName:f379b9b0-301a-47af-8703-b07da27e797e nodeName:}" failed. No retries permitted until 2026-09-09 20:11:27.082551284 +0000 UTC m=+3.056321348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qh76s" (UniqueName: "kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s") pod "network-check-target-sdx6r" (UID: "f379b9b0-301a-47af-8703-b07da27e797e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:26.584413 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.584364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lslt\" (UniqueName: \"kubernetes.io/projected/cea6223c-9ead-467f-b0e8-c09ceb9ec51a-kube-api-access-4lslt\") pod \"node-resolver-2cdv6\" (UID: \"cea6223c-9ead-467f-b0e8-c09ceb9ec51a\") " pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:26.584516 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.584448    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllqt\" (UniqueName: \"kubernetes.io/projected/b355c14c-96e4-4d5d-ada5-6142cd99e951-kube-api-access-vllqt\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:26.584726 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.584705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqm9\" (UniqueName: \"kubernetes.io/projected/8335a8ba-a535-4f1b-b63d-285c7107f06c-kube-api-access-tcqm9\") pod \"tuned-7dzkh\" (UID: \"8335a8ba-a535-4f1b-b63d-285c7107f06c\") " pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.585613 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.585590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl75r\" (UniqueName: \"kubernetes.io/projected/6e1cd48d-d40b-4a81-beb9-5a62463c96ca-kube-api-access-fl75r\") pod \"iptables-alerter-g24n5\" (UID: \"6e1cd48d-d40b-4a81-beb9-5a62463c96ca\") " pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.585704 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.585676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qng8c\" (UniqueName: \"kubernetes.io/projected/84e0d8c1-8676-409f-96e7-ff26d02cd1f7-kube-api-access-qng8c\") pod \"ovnkube-node-xt4pz\" (UID: \"84e0d8c1-8676-409f-96e7-ff26d02cd1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.622065 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.621890    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:26.666401 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.666373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:26.675686 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.675667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6pfl"
Sep 09 20:11:26.683505 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.683485    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8"
Sep 09 20:11:26.691336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.691310    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bm5wh"
Sep 09 20:11:26.698890 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.698875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k5bpz"
Sep 09 20:11:26.707788 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.707770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g24n5"
Sep 09 20:11:26.716616 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.716595    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:26.722156 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.722138    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7dzkh"
Sep 09 20:11:26.727648 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:26.727632    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2cdv6"
Sep 09 20:11:27.079044 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.079012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:27.079218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.079089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:27.079218 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.079129    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:27.079218 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.079199    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:28.079180721 +0000 UTC m=+4.052950775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:27.079218 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.079200    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:27.079422 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.079253    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret podName:a80d1370-4cbe-4f65-8dd5-d42457f347b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:28.07923695 +0000 UTC m=+4.053007016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret") pod "global-pull-secret-syncer-qkfk2" (UID: "a80d1370-4cbe-4f65-8dd5-d42457f347b2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:27.179726 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.179693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:27.179908 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.179883    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:27.179908 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.179908    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:27.180025 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.179921    2562 projected.go:196] Error preparing data for projected volume kube-api-access-qh76s for pod openshift-network-diagnostics/network-check-target-sdx6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:27.180025 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.179988    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s podName:f379b9b0-301a-47af-8703-b07da27e797e nodeName:}" failed. No retries permitted until 2026-09-09 20:11:28.179967984 +0000 UTC m=+4.153738038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qh76s" (UniqueName: "kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s") pod "network-check-target-sdx6r" (UID: "f379b9b0-301a-47af-8703-b07da27e797e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:27.277041 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.277016    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea6223c_9ead_467f_b0e8_c09ceb9ec51a.slice/crio-9251c115856168574c77f5c75522806c0f02fefa2876d20711dd7e5f57ae411a WatchSource:0}: Error finding container 9251c115856168574c77f5c75522806c0f02fefa2876d20711dd7e5f57ae411a: Status 404 returned error can't find the container with id 9251c115856168574c77f5c75522806c0f02fefa2876d20711dd7e5f57ae411a
Sep 09 20:11:27.279263 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.279232    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e0d8c1_8676_409f_96e7_ff26d02cd1f7.slice/crio-f4027c09969001eb98cc666117b63135f4d930268860d18146d90d8417510ec1 WatchSource:0}: Error finding container f4027c09969001eb98cc666117b63135f4d930268860d18146d90d8417510ec1: Status 404 returned error can't find the container with id f4027c09969001eb98cc666117b63135f4d930268860d18146d90d8417510ec1
Sep 09 20:11:27.280248 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.280222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040eebcb_98c6_4c25_99f3_c9a0f400a644.slice/crio-d12aea61ae03be5f2f22ae831adc9d2dcd5c5e33c27fb9fe0075d77261406585 WatchSource:0}: Error finding container d12aea61ae03be5f2f22ae831adc9d2dcd5c5e33c27fb9fe0075d77261406585: Status 404 returned error can't find the container with id d12aea61ae03be5f2f22ae831adc9d2dcd5c5e33c27fb9fe0075d77261406585
Sep 09 20:11:27.280550 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.280527    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512ceb34_1d30_47d0_b1f4_e2a0871e519b.slice/crio-e0f5a841ecfce7f7fc5283af5e18e9d50b47f7ef59308412eca4b05117849c74 WatchSource:0}: Error finding container e0f5a841ecfce7f7fc5283af5e18e9d50b47f7ef59308412eca4b05117849c74: Status 404 returned error can't find the container with id e0f5a841ecfce7f7fc5283af5e18e9d50b47f7ef59308412eca4b05117849c74
Sep 09 20:11:27.282240 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.282014    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8335a8ba_a535_4f1b_b63d_285c7107f06c.slice/crio-113d56bcc1004703b991a609695043ad4bf60376addeefbb2d0a31aba0d7ae8c WatchSource:0}: Error finding container 113d56bcc1004703b991a609695043ad4bf60376addeefbb2d0a31aba0d7ae8c: Status 404 returned error can't find the container with id 113d56bcc1004703b991a609695043ad4bf60376addeefbb2d0a31aba0d7ae8c
Sep 09 20:11:27.284879 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.284827    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0762481_ab10_46dd_a065_727d330ce5e0.slice/crio-acb44f949838c2287aff00c51666c52aa7278b2f58dbbc2c55d85fa0a81cffd7 WatchSource:0}: Error finding container acb44f949838c2287aff00c51666c52aa7278b2f58dbbc2c55d85fa0a81cffd7: Status 404 returned error can't find the container with id acb44f949838c2287aff00c51666c52aa7278b2f58dbbc2c55d85fa0a81cffd7
Sep 09 20:11:27.285545 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.285523    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1433c535_6ce0_4f9b_b94c_a42e534ff3ee.slice/crio-895fbd048c6178a6252ea3295dc8e756d3cee172bcbd5bf76847180085f0c316 WatchSource:0}: Error finding container 895fbd048c6178a6252ea3295dc8e756d3cee172bcbd5bf76847180085f0c316: Status 404 returned error can't find the container with id 895fbd048c6178a6252ea3295dc8e756d3cee172bcbd5bf76847180085f0c316
Sep 09 20:11:27.292407 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:27.292288    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc038dca1_e9c9_4c1a_aefb_4125c06f7371.slice/crio-63254a28f1e9d18a4d839031239d920367c83571b880b1c54b870bb12f3e8a18 WatchSource:0}: Error finding container 63254a28f1e9d18a4d839031239d920367c83571b880b1c54b870bb12f3e8a18: Status 404 returned error can't find the container with id 63254a28f1e9d18a4d839031239d920367c83571b880b1c54b870bb12f3e8a18
Sep 09 20:11:27.403190 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.403105    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:26 +0000 UTC" deadline="2028-03-14 12:40:36.000054211 +0000 UTC"
Sep 09 20:11:27.403190 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.403157    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13240h29m8.596900619s"
Sep 09 20:11:27.460650 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.460628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:27.460756 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:27.460719    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:27.470885 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.470861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerStarted","Data":"63254a28f1e9d18a4d839031239d920367c83571b880b1c54b870bb12f3e8a18"}
Sep 09 20:11:27.473023 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.472983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g24n5" event={"ID":"6e1cd48d-d40b-4a81-beb9-5a62463c96ca","Type":"ContainerStarted","Data":"ab59d937313fda4bb410595c42f435b7e7f7798b11cdd40c11210c4163494778"}
Sep 09 20:11:27.474198 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.474177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6pfl" event={"ID":"512ceb34-1d30-47d0-b1f4-e2a0871e519b","Type":"ContainerStarted","Data":"e0f5a841ecfce7f7fc5283af5e18e9d50b47f7ef59308412eca4b05117849c74"}
Sep 09 20:11:27.476980 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.476955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8" event={"ID":"040eebcb-98c6-4c25-99f3-c9a0f400a644","Type":"ContainerStarted","Data":"d12aea61ae03be5f2f22ae831adc9d2dcd5c5e33c27fb9fe0075d77261406585"}
Sep 09 20:11:27.478729 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.478697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"f4027c09969001eb98cc666117b63135f4d930268860d18146d90d8417510ec1"}
Sep 09 20:11:27.479856 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.479835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lkdsq" event={"ID":"1433c535-6ce0-4f9b-b94c-a42e534ff3ee","Type":"ContainerStarted","Data":"895fbd048c6178a6252ea3295dc8e756d3cee172bcbd5bf76847180085f0c316"}
Sep 09 20:11:27.480859 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.480841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k5bpz" event={"ID":"b0762481-ab10-46dd-a065-727d330ce5e0","Type":"ContainerStarted","Data":"acb44f949838c2287aff00c51666c52aa7278b2f58dbbc2c55d85fa0a81cffd7"}
Sep 09 20:11:27.481947 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.481918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7dzkh" event={"ID":"8335a8ba-a535-4f1b-b63d-285c7107f06c","Type":"ContainerStarted","Data":"113d56bcc1004703b991a609695043ad4bf60376addeefbb2d0a31aba0d7ae8c"}
Sep 09 20:11:27.483112 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.483068    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2cdv6" event={"ID":"cea6223c-9ead-467f-b0e8-c09ceb9ec51a","Type":"ContainerStarted","Data":"9251c115856168574c77f5c75522806c0f02fefa2876d20711dd7e5f57ae411a"}
Sep 09 20:11:27.485403 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.485385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal" event={"ID":"2f674f484fa389b34823ea538de3d17f","Type":"ContainerStarted","Data":"83a0ec652eb4568dee998c91e8d7183983bcad29c64ff30084650bc7f7c4f0ec"}
Sep 09 20:11:27.501319 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:27.501275    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal" podStartSLOduration=2.50126287 podStartE2EDuration="2.50126287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:27.501130001 +0000 UTC m=+3.474900068" watchObservedRunningTime="2026-09-09 20:11:27.50126287 +0000 UTC m=+3.475032912"
Sep 09 20:11:28.087542 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.087503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:28.087718 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.087590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:28.087795 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.087730    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:28.087846 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.087797    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret podName:a80d1370-4cbe-4f65-8dd5-d42457f347b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:30.087778317 +0000 UTC m=+6.061548380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret") pod "global-pull-secret-syncer-qkfk2" (UID: "a80d1370-4cbe-4f65-8dd5-d42457f347b2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:28.088493 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.088472    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:28.088593 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.088535    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:30.088516108 +0000 UTC m=+6.062286191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:28.188915 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.188871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:28.189115 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.189096    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:28.189180 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.189122    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:28.189180 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.189136    2562 projected.go:196] Error preparing data for projected volume kube-api-access-qh76s for pod openshift-network-diagnostics/network-check-target-sdx6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:28.189288 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.189198    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s podName:f379b9b0-301a-47af-8703-b07da27e797e nodeName:}" failed. No retries permitted until 2026-09-09 20:11:30.189179395 +0000 UTC m=+6.162949438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qh76s" (UniqueName: "kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s") pod "network-check-target-sdx6r" (UID: "f379b9b0-301a-47af-8703-b07da27e797e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:28.404266 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.403910    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:26 +0000 UTC" deadline="2028-05-01 15:54:27.044131137 +0000 UTC"
Sep 09 20:11:28.404266 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.403948    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14395h42m58.640188051s"
Sep 09 20:11:28.472898 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.472870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:28.473054 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.473021    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:28.473430 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.473409    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:28.473533 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:28.473508    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:28.509995 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.509962    2562 generic.go:356] "Generic (PLEG): container finished" podID="c6277271f02f9fd3891a478b286fe5b5" containerID="f923f458db3ed6d9a4d0efaea124a54384225d791846920c4823db74f06bdfad" exitCode=0
Sep 09 20:11:28.510905 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:28.510877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" event={"ID":"c6277271f02f9fd3891a478b286fe5b5","Type":"ContainerDied","Data":"f923f458db3ed6d9a4d0efaea124a54384225d791846920c4823db74f06bdfad"}
Sep 09 20:11:29.460046 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:29.459955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:29.460682 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:29.460131    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:29.528032 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:29.527829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" event={"ID":"c6277271f02f9fd3891a478b286fe5b5","Type":"ContainerStarted","Data":"f0a5597a242032a21f928a379c7aefabd5d41f6431938123c8ae3dbe5bb5532a"}
Sep 09 20:11:29.546751 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:29.545542    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" podStartSLOduration=4.545521363 podStartE2EDuration="4.545521363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.545383517 +0000 UTC m=+5.519153581" watchObservedRunningTime="2026-09-09 20:11:29.545521363 +0000 UTC m=+5.519291421"
Sep 09 20:11:30.107928 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:30.107883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:30.108123 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:30.107971    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:30.108191 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.108128    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:30.108191 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.108189    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret podName:a80d1370-4cbe-4f65-8dd5-d42457f347b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:34.108171099 +0000 UTC m=+10.081941154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret") pod "global-pull-secret-syncer-qkfk2" (UID: "a80d1370-4cbe-4f65-8dd5-d42457f347b2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:30.108622 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.108602    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:30.108719 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.108656    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:34.108640866 +0000 UTC m=+10.082410922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:30.209166 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:30.209121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:30.209354 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.209332    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:30.209416 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.209363    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:30.209416 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.209377    2562 projected.go:196] Error preparing data for projected volume kube-api-access-qh76s for pod openshift-network-diagnostics/network-check-target-sdx6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:30.209523 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.209442    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s podName:f379b9b0-301a-47af-8703-b07da27e797e nodeName:}" failed. No retries permitted until 2026-09-09 20:11:34.209423995 +0000 UTC m=+10.183194040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qh76s" (UniqueName: "kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s") pod "network-check-target-sdx6r" (UID: "f379b9b0-301a-47af-8703-b07da27e797e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:30.461179 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:30.461107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:30.461607 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.461238    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:30.461821 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:30.461788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:30.461921 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:30.461899    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:31.460581 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:31.460105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:31.460581 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:31.460242    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:32.461077 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:32.460585    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:32.461077 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:32.460715    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:32.461077 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:32.460789    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:32.461077 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:32.460916    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:33.460552 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:33.460518    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:33.460729 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:33.460667    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:34.147026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:34.146908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:34.147026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:34.146983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:34.147497 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.147098    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:34.147497 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.147100    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:34.147497 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.147171    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret podName:a80d1370-4cbe-4f65-8dd5-d42457f347b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.147148252 +0000 UTC m=+18.120918294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret") pod "global-pull-secret-syncer-qkfk2" (UID: "a80d1370-4cbe-4f65-8dd5-d42457f347b2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:34.147497 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.147192    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.147178374 +0000 UTC m=+18.120948434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:34.247942 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:34.247906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:34.248154 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.248117    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:34.248154 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.248140    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:34.248154 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.248153    2562 projected.go:196] Error preparing data for projected volume kube-api-access-qh76s for pod openshift-network-diagnostics/network-check-target-sdx6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:34.248331 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.248211    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s podName:f379b9b0-301a-47af-8703-b07da27e797e nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.248193645 +0000 UTC m=+18.221963689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qh76s" (UniqueName: "kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s") pod "network-check-target-sdx6r" (UID: "f379b9b0-301a-47af-8703-b07da27e797e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:34.461225 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:34.461094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:34.461370 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.461222    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:34.461370 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:34.461273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:34.461370 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:34.461336    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:35.459916 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:35.459664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:35.460370 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:35.459945    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:36.460060 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:36.460034    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:36.460497 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:36.460066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:36.460497 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:36.460156    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:36.460497 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:36.460268    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:37.460208 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:37.460175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:37.460540 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:37.460303    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:38.460359 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:38.460322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:38.460740 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:38.460360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:38.460740 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:38.460478    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:38.460740 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:38.460603    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:39.460536 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:39.460501    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:39.460966 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:39.460607    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:40.460689 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:40.460653    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:40.461113 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:40.460703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:40.461113 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:40.460797    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:40.461113 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:40.460933    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:41.460248 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:41.460218    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:41.460522 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:41.460342    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:42.204272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:42.204231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:42.204746 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:42.204319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:42.204746 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.204355    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:42.204746 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.204420    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:42.204746 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.204429    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:58.20440968 +0000 UTC m=+34.178179724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:42.204746 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.204465    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret podName:a80d1370-4cbe-4f65-8dd5-d42457f347b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:58.20445022 +0000 UTC m=+34.178220277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret") pod "global-pull-secret-syncer-qkfk2" (UID: "a80d1370-4cbe-4f65-8dd5-d42457f347b2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:42.304626 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:42.304587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:42.304804 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.304779    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:42.304804 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.304802    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:42.304910 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.304815    2562 projected.go:196] Error preparing data for projected volume kube-api-access-qh76s for pod openshift-network-diagnostics/network-check-target-sdx6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:42.304910 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.304881    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s podName:f379b9b0-301a-47af-8703-b07da27e797e nodeName:}" failed. No retries permitted until 2026-09-09 20:11:58.304862279 +0000 UTC m=+34.278632344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qh76s" (UniqueName: "kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s") pod "network-check-target-sdx6r" (UID: "f379b9b0-301a-47af-8703-b07da27e797e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:42.460567 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:42.460487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:42.460567 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:42.460535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:42.460778 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.460630    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:42.460778 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:42.460754    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:43.459589 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:43.459559    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:43.460036 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:43.459661    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:44.460689 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:44.460658    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:44.460689 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:44.460685    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:44.461204 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:44.460762    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:44.461204 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:44.460868    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:45.460763 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.460599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:45.461237 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:45.460839    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:45.559611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.559541    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lkdsq" event={"ID":"1433c535-6ce0-4f9b-b94c-a42e534ff3ee","Type":"ContainerStarted","Data":"437404fb8833c797ecde19fa8abeea2be4602e0f2549d2e6b4a6d68e90d7f96a"}
Sep 09 20:11:45.561141 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.561117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k5bpz" event={"ID":"b0762481-ab10-46dd-a065-727d330ce5e0","Type":"ContainerStarted","Data":"20b1ba1f61f13c4d15f903a492bf7705c51de6d472689437e562ed6fe62c620c"}
Sep 09 20:11:45.562625 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.562603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7dzkh" event={"ID":"8335a8ba-a535-4f1b-b63d-285c7107f06c","Type":"ContainerStarted","Data":"68a39a70a2a2777ad6484a07a2b362b300944c95f45092055ef729d68ff49af6"}
Sep 09 20:11:45.564206 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.564180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2cdv6" event={"ID":"cea6223c-9ead-467f-b0e8-c09ceb9ec51a","Type":"ContainerStarted","Data":"9588010b158e82af3638e1076f1f4425947362025cace27c97f97f7b82206ff1"}
Sep 09 20:11:45.565623 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.565601    2562 generic.go:356] "Generic (PLEG): container finished" podID="c038dca1-e9c9-4c1a-aefb-4125c06f7371" containerID="f85cbeece21afefce93737fbdab8a2b3671e97618f1537ecb11cc1d5b8a30952" exitCode=0
Sep 09 20:11:45.565701 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.565675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerDied","Data":"f85cbeece21afefce93737fbdab8a2b3671e97618f1537ecb11cc1d5b8a30952"}
Sep 09 20:11:45.567074 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.567052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6pfl" event={"ID":"512ceb34-1d30-47d0-b1f4-e2a0871e519b","Type":"ContainerStarted","Data":"3267f111f9abd11bed17cb87fdfeb866839d0dd5d2aebfc366636b22e96f6c4b"}
Sep 09 20:11:45.568447 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.568429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8" event={"ID":"040eebcb-98c6-4c25-99f3-c9a0f400a644","Type":"ContainerStarted","Data":"d6d740e1b3bdf4af194192131577a21f8991ce24df9a52f9090fd6e5f0e6c955"}
Sep 09 20:11:45.570614 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.570596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"72c39cab75bb6914cfff40fcf1fb8e00b910107a1de2f8f27c24db6f8726f34d"}
Sep 09 20:11:45.570681 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.570647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"e8f5871e53e1b4f83be0cc8778ed956105cb2bb7ed19009e6e26246973525d02"}
Sep 09 20:11:45.570681 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.570678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"b1b8d77876145408fe46eb73ffd8fa9453f3b62493d5fef6a57146b641b911d2"}
Sep 09 20:11:45.570796 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.570715    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"3f1219c61d2ed125e501238868f8bd7dec4e134b9b012758d93c2634d0874d4e"}
Sep 09 20:11:45.575093 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.575058    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lkdsq" podStartSLOduration=3.977651378 podStartE2EDuration="21.575039604s" totalImagesPullingTime="17.597388226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:45.574692763 +0000 UTC m=+21.548462838" watchObservedRunningTime="2026-09-09 20:11:45.575039604 +0000 UTC m=+21.548809680"
Sep 09 20:11:45.614994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.614958    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k5bpz" podStartSLOduration=3.839213546 podStartE2EDuration="21.614946652s" totalImagesPullingTime="17.775733106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:45.614515222 +0000 UTC m=+21.588285302" watchObservedRunningTime="2026-09-09 20:11:45.614946652 +0000 UTC m=+21.588716711"
Sep 09 20:11:45.632440 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.632381    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7dzkh" podStartSLOduration=3.8924696130000003 podStartE2EDuration="21.63236805s" totalImagesPullingTime="17.739898437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:45.632125113 +0000 UTC m=+21.605895202" watchObservedRunningTime="2026-09-09 20:11:45.63236805 +0000 UTC m=+21.606138128"
Sep 09 20:11:45.646424 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.646392    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n6pfl" podStartSLOduration=11.345097413 podStartE2EDuration="21.646377112s" totalImagesPullingTime="10.301279699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:45.645904176 +0000 UTC m=+21.619674239" watchObservedRunningTime="2026-09-09 20:11:45.646377112 +0000 UTC m=+21.620147176"
Sep 09 20:11:45.659753 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:45.659722    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2cdv6" podStartSLOduration=4.052672031 podStartE2EDuration="21.659710553s" totalImagesPullingTime="17.607038522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:45.659159966 +0000 UTC m=+21.632930030" watchObservedRunningTime="2026-09-09 20:11:45.659710553 +0000 UTC m=+21.633480617"
Sep 09 20:11:46.450491 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.450464    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:46.459929 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.459893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:46.460064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.459895    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:46.460064 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:46.460044    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:46.460179 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:46.460109    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:46.574603 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.574395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g24n5" event={"ID":"6e1cd48d-d40b-4a81-beb9-5a62463c96ca","Type":"ContainerStarted","Data":"da09e523e79b9f8bcc3b3962207b05ad12b5bf723982115f168f4e0285aeee42"}
Sep 09 20:11:46.576576 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.576542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8" event={"ID":"040eebcb-98c6-4c25-99f3-c9a0f400a644","Type":"ContainerStarted","Data":"b5b0b688237f1d0005eb6701aeeafb5f119fd16809834bd40ccb4cb45a050ee6"}
Sep 09 20:11:46.580267 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.580239    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"ce426f175bbeb0f8a373ad255e688a52003c5e2dbd4e3049be64d45b9008e0ef"}
Sep 09 20:11:46.580401 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.580310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"2242cca63e7b3a8718613a70cb34623dbde6e1b8a7e671eb01990a04a7d7cdf6"}
Sep 09 20:11:46.588514 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:46.588475    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g24n5" podStartSLOduration=4.983008998 podStartE2EDuration="22.588459748s" totalImagesPullingTime="17.60545075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:46.588240238 +0000 UTC m=+22.562010306" watchObservedRunningTime="2026-09-09 20:11:46.588459748 +0000 UTC m=+22.562229812"
Sep 09 20:11:47.420909 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:47.420799    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:46.450486041Z","UUID":"fb465dcb-bfe5-420f-bb53-cfd0d42af2ae","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:47.423739 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:47.423711    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:47.423871 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:47.423748    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:47.459866 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:47.459844    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:47.459992 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:47.459964    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:47.584915 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:47.584877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8" event={"ID":"040eebcb-98c6-4c25-99f3-c9a0f400a644","Type":"ContainerStarted","Data":"c921062ab632a0a15f9a75c45cf4a162dff4ab9f67d57d75c255e0810d361ce6"}
Sep 09 20:11:47.602198 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:47.602151    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fmz8" podStartSLOduration=3.947532838 podStartE2EDuration="23.602130269s" totalImagesPullingTime="19.654597431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:47.601564584 +0000 UTC m=+23.575334660" watchObservedRunningTime="2026-09-09 20:11:47.602130269 +0000 UTC m=+23.575900340"
Sep 09 20:11:48.460688 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:48.460655    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:48.460955 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:48.460775    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:48.460955 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:48.460826    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:48.460955 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:48.460937    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:48.590814 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:48.590774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"5c433a67206f6dc326e3681f44ed4698cb237e0859f4f81172303680480db36a"}
Sep 09 20:11:49.460118 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:49.460084    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:49.460289 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:49.460209    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:50.325519 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.325352    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:50.325854 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.325833    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:50.462773 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.462710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:50.462773 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.462723    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:50.462909 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:50.462811    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:50.462941 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:50.462926    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:50.595920 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.595891    2562 generic.go:356] "Generic (PLEG): container finished" podID="c038dca1-e9c9-4c1a-aefb-4125c06f7371" containerID="083f2e4fb81c93b2ebb64abd98a601c64a3ecbe48bd361f887fdf0b3237e4eda" exitCode=0
Sep 09 20:11:50.596144 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.595980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerDied","Data":"083f2e4fb81c93b2ebb64abd98a601c64a3ecbe48bd361f887fdf0b3237e4eda"}
Sep 09 20:11:50.599992 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.599969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" event={"ID":"84e0d8c1-8676-409f-96e7-ff26d02cd1f7","Type":"ContainerStarted","Data":"6f0c15a35807248dd3f1f2edf591739a309822e1c243612f2275311c48c984b4"}
Sep 09 20:11:50.600225 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.600201    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:50.600552 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.600538    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lkdsq"
Sep 09 20:11:50.654326 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:50.654283    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz" podStartSLOduration=8.796708819 podStartE2EDuration="26.654271422s" totalImagesPullingTime="17.857562603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:50.652770226 +0000 UTC m=+26.626540289" watchObservedRunningTime="2026-09-09 20:11:50.654271422 +0000 UTC m=+26.628041493"
Sep 09 20:11:51.460503 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.460474    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:51.460928 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:51.460576    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:51.602369 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.602342    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:51.602533 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.602424    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:51.602533 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.602476    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:51.616682 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.616657    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:51.616921 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.616906    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:11:51.999532 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.999485    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdx6r"]
Sep 09 20:11:51.999850 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:51.999719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:52.000171 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:52.000137    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:52.000251 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:52.000224    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkfk2"]
Sep 09 20:11:52.000442 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:52.000322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:52.000442 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:52.000407    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:52.001868 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:52.001588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zq8t9"]
Sep 09 20:11:52.001868 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:52.001705    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:52.001868 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:52.001833    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:52.605885 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:52.605801    2562 generic.go:356] "Generic (PLEG): container finished" podID="c038dca1-e9c9-4c1a-aefb-4125c06f7371" containerID="2fcf2029f279ca5a429993789830558e96939e4af4a26fb5651190e7c2f7c94b" exitCode=0
Sep 09 20:11:52.606524 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:52.605878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerDied","Data":"2fcf2029f279ca5a429993789830558e96939e4af4a26fb5651190e7c2f7c94b"}
Sep 09 20:11:53.460109 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:53.459879    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:53.460216 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:53.459877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:53.460216 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:53.460145    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:53.460216 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:53.459879    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:53.460301 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:53.460214    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:53.460344 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:53.460328    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:53.613463 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:53.613426    2562 generic.go:356] "Generic (PLEG): container finished" podID="c038dca1-e9c9-4c1a-aefb-4125c06f7371" containerID="bfca50490c259fd7e8446cd4fe91c099d4c5df69b803036fb151c520ab67d545" exitCode=0
Sep 09 20:11:53.613847 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:53.613499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerDied","Data":"bfca50490c259fd7e8446cd4fe91c099d4c5df69b803036fb151c520ab67d545"}
Sep 09 20:11:55.459621 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:55.459591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:55.460064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:55.459634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:55.460064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:55.459591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:55.460064 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:55.459708    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:55.460064 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:55.459764    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:55.460064 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:55.459838    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:57.460511 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.460478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:57.461098 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.460522    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:57.461098 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.460482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:57.461098 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:57.460614    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:11:57.461098 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:57.460696    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkfk2" podUID="a80d1370-4cbe-4f65-8dd5-d42457f347b2"
Sep 09 20:11:57.461098 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:57.460778    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdx6r" podUID="f379b9b0-301a-47af-8703-b07da27e797e"
Sep 09 20:11:57.880055 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.880031    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeReady"
Sep 09 20:11:57.880220 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.880174    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:11:57.928720 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.928687    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jghvp"]
Sep 09 20:11:57.984975 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.984942    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s5bqc"]
Sep 09 20:11:57.985161 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.985069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:57.988106 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.987907    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:11:57.988106 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.987925    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nht2s\""
Sep 09 20:11:57.988106 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.988044    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:11:57.997883 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.997864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s5bqc"]
Sep 09 20:11:57.997986 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.997894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jghvp"]
Sep 09 20:11:57.997986 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:57.997976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.001058 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.001033    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:11:58.001205 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.001056    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lzq8l\""
Sep 09 20:11:58.001205 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.001039    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:11:58.001205 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.001111    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:11:58.121079 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.121047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6313d4f-b898-4da8-8010-c67a8726dfea-tmp-dir\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.121207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.121116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66c4d70d-5928-4cc4-82c6-3179c6999aa7-cert\") pod \"ingress-canary-s5bqc\" (UID: \"66c4d70d-5928-4cc4-82c6-3179c6999aa7\") " pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.121207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.121189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kx6t\" (UniqueName: \"kubernetes.io/projected/66c4d70d-5928-4cc4-82c6-3179c6999aa7-kube-api-access-8kx6t\") pod \"ingress-canary-s5bqc\" (UID: \"66c4d70d-5928-4cc4-82c6-3179c6999aa7\") " pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.121303 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.121208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.121303 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.121232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6313d4f-b898-4da8-8010-c67a8726dfea-config-volume\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.121303 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.121251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j89pn\" (UniqueName: \"kubernetes.io/projected/a6313d4f-b898-4da8-8010-c67a8726dfea-kube-api-access-j89pn\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.221853 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.221822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6313d4f-b898-4da8-8010-c67a8726dfea-tmp-dir\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.221992 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.221864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:58.221992 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.221887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66c4d70d-5928-4cc4-82c6-3179c6999aa7-cert\") pod \"ingress-canary-s5bqc\" (UID: \"66c4d70d-5928-4cc4-82c6-3179c6999aa7\") " pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.221992 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.221925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.221994    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.222037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kx6t\" (UniqueName: \"kubernetes.io/projected/66c4d70d-5928-4cc4-82c6-3179c6999aa7-kube-api-access-8kx6t\") pod \"ingress-canary-s5bqc\" (UID: \"66c4d70d-5928-4cc4-82c6-3179c6999aa7\") " pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.222059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.222073    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:30.222054776 +0000 UTC m=+66.195824837 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.222107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6313d4f-b898-4da8-8010-c67a8726dfea-config-volume\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.222016    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.222125    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.222131    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89pn\" (UniqueName: \"kubernetes.io/projected/a6313d4f-b898-4da8-8010-c67a8726dfea-kube-api-access-j89pn\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.222168    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret podName:a80d1370-4cbe-4f65-8dd5-d42457f347b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:30.222149204 +0000 UTC m=+66.195919259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret") pod "global-pull-secret-syncer-qkfk2" (UID: "a80d1370-4cbe-4f65-8dd5-d42457f347b2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.222187    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:11:58.722177435 +0000 UTC m=+34.695947477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:58.222234 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.222220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6313d4f-b898-4da8-8010-c67a8726dfea-tmp-dir\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.222591 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.222551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6313d4f-b898-4da8-8010-c67a8726dfea-config-volume\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.226064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.226039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66c4d70d-5928-4cc4-82c6-3179c6999aa7-cert\") pod \"ingress-canary-s5bqc\" (UID: \"66c4d70d-5928-4cc4-82c6-3179c6999aa7\") " pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.243098 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.243075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89pn\" (UniqueName: \"kubernetes.io/projected/a6313d4f-b898-4da8-8010-c67a8726dfea-kube-api-access-j89pn\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.243368 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.243352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kx6t\" (UniqueName: \"kubernetes.io/projected/66c4d70d-5928-4cc4-82c6-3179c6999aa7-kube-api-access-8kx6t\") pod \"ingress-canary-s5bqc\" (UID: \"66c4d70d-5928-4cc4-82c6-3179c6999aa7\") " pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.308076 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.308056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s5bqc"
Sep 09 20:11:58.325040 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.322770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:58.325040 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.322932    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:58.325040 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.322950    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:58.325040 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.322970    2562 projected.go:196] Error preparing data for projected volume kube-api-access-qh76s for pod openshift-network-diagnostics/network-check-target-sdx6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:58.325040 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.323058    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s podName:f379b9b0-301a-47af-8703-b07da27e797e nodeName:}" failed. No retries permitted until 2026-09-09 20:12:30.323033913 +0000 UTC m=+66.296803956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qh76s" (UniqueName: "kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s") pod "network-check-target-sdx6r" (UID: "f379b9b0-301a-47af-8703-b07da27e797e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:58.628279 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.628027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerStarted","Data":"dd5a23aacca7eaa1bd5529ff9d9926e52d893b03edce49cde4b83889d912b8c0"}
Sep 09 20:11:58.669425 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.668887    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s5bqc"]
Sep 09 20:11:58.692789 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:11:58.692763    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c4d70d_5928_4cc4_82c6_3179c6999aa7.slice/crio-24d8ec61f9286e2a479278db08215bf8e26090f41eaef8604b2abc81f1520075 WatchSource:0}: Error finding container 24d8ec61f9286e2a479278db08215bf8e26090f41eaef8604b2abc81f1520075: Status 404 returned error can't find the container with id 24d8ec61f9286e2a479278db08215bf8e26090f41eaef8604b2abc81f1520075
Sep 09 20:11:58.726442 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:58.726411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:58.726545 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.726528    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:58.726603 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:58.726598    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:11:59.726580828 +0000 UTC m=+35.700350882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:59.460107 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.460076    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:11:59.460282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.460076    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:11:59.460347 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.460083    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:11:59.465599 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.465517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hb455\""
Sep 09 20:11:59.466755 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.466731    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:11:59.466755 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.466752    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qg729\""
Sep 09 20:11:59.466930 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.466732    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:11:59.466930 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.466758    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:11:59.466930 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.466731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:11:59.633522 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.633457    2562 generic.go:356] "Generic (PLEG): container finished" podID="c038dca1-e9c9-4c1a-aefb-4125c06f7371" containerID="dd5a23aacca7eaa1bd5529ff9d9926e52d893b03edce49cde4b83889d912b8c0" exitCode=0
Sep 09 20:11:59.634102 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.633540    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerDied","Data":"dd5a23aacca7eaa1bd5529ff9d9926e52d893b03edce49cde4b83889d912b8c0"}
Sep 09 20:11:59.635017 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.634981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s5bqc" event={"ID":"66c4d70d-5928-4cc4-82c6-3179c6999aa7","Type":"ContainerStarted","Data":"24d8ec61f9286e2a479278db08215bf8e26090f41eaef8604b2abc81f1520075"}
Sep 09 20:11:59.732993 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:11:59.732965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:11:59.733265 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:59.733244    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:59.733375 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:11:59.733303    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:12:01.733288078 +0000 UTC m=+37.707058124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:00.641321 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:00.641285    2562 generic.go:356] "Generic (PLEG): container finished" podID="c038dca1-e9c9-4c1a-aefb-4125c06f7371" containerID="b0cc9c59994091df83d31418b3945c7652c76b66b6813a334408aa61ae1f4e80" exitCode=0
Sep 09 20:12:00.641321 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:00.641329    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerDied","Data":"b0cc9c59994091df83d31418b3945c7652c76b66b6813a334408aa61ae1f4e80"}
Sep 09 20:12:01.647278 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:01.647243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" event={"ID":"c038dca1-e9c9-4c1a-aefb-4125c06f7371","Type":"ContainerStarted","Data":"51112680c861dfc4929dd125e5de3d89179f21779869075c41f8d7d15f781817"}
Sep 09 20:12:01.648667 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:01.648643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s5bqc" event={"ID":"66c4d70d-5928-4cc4-82c6-3179c6999aa7","Type":"ContainerStarted","Data":"a699031a838e7cb0a5b2c600ee240e7d09637c3bc74b5cf3c679e9983770a7cc"}
Sep 09 20:12:01.670305 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:01.670256    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bm5wh" podStartSLOduration=8.457410227 podStartE2EDuration="37.670242743s" totalImagesPullingTime="29.212832516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:01.668973509 +0000 UTC m=+37.642743566" watchObservedRunningTime="2026-09-09 20:12:01.670242743 +0000 UTC m=+37.644012803"
Sep 09 20:12:01.684164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:01.684121    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s5bqc" podStartSLOduration=2.287813289 podStartE2EDuration="4.684106603s" totalImagesPullingTime="2.396293314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:01.683274114 +0000 UTC m=+37.657044189" watchObservedRunningTime="2026-09-09 20:12:01.684106603 +0000 UTC m=+37.657876666"
Sep 09 20:12:01.747519 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:01.747481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:12:01.747700 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:01.747595    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:01.747764 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:01.747720    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.747695006 +0000 UTC m=+41.721465064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:05.778661 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:05.778614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:12:05.779138 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:05.778769    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:05.779138 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:05.778829    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:12:13.77881191 +0000 UTC m=+49.752581970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:13.836835 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:13.836786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:12:13.837330 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:13.836930    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:13.837330 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:13.837019    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:12:29.836988658 +0000 UTC m=+65.810758698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:23.623656 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:23.623627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xt4pz"
Sep 09 20:12:29.843513 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:29.843472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:12:29.843889 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:29.843621    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:29.843889 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:29.843684    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:13:01.843667711 +0000 UTC m=+97.817437756 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:30.245256 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.245214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:12:30.245442 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.245277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:12:30.247915 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.247894    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:30.248053 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.248036    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:30.256083 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:30.256062    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:12:30.256160 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:12:30.256118    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:13:34.256103402 +0000 UTC m=+130.229873448 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : secret "metrics-daemon-secret" not found
Sep 09 20:12:30.259059 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.259041    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a80d1370-4cbe-4f65-8dd5-d42457f347b2-original-pull-secret\") pod \"global-pull-secret-syncer-qkfk2\" (UID: \"a80d1370-4cbe-4f65-8dd5-d42457f347b2\") " pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:12:30.345514 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.345476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:12:30.348201 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.348182    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:12:30.358092 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.358078    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:12:30.369288 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.369269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh76s\" (UniqueName: \"kubernetes.io/projected/f379b9b0-301a-47af-8703-b07da27e797e-kube-api-access-qh76s\") pod \"network-check-target-sdx6r\" (UID: \"f379b9b0-301a-47af-8703-b07da27e797e\") " pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:12:30.382571 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.382550    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qg729\""
Sep 09 20:12:30.384556 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.384544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkfk2"
Sep 09 20:12:30.391026 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.390989    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:12:30.658333 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.658304    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkfk2"]
Sep 09 20:12:30.661151 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.661111    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdx6r"]
Sep 09 20:12:30.661351 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:12:30.661332    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80d1370_4cbe_4f65_8dd5_d42457f347b2.slice/crio-ca981e993b20d18d249a8556f612f396e8d7e1bab0041ca7c6b003c505c09bed WatchSource:0}: Error finding container ca981e993b20d18d249a8556f612f396e8d7e1bab0041ca7c6b003c505c09bed: Status 404 returned error can't find the container with id ca981e993b20d18d249a8556f612f396e8d7e1bab0041ca7c6b003c505c09bed
Sep 09 20:12:30.664158 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:12:30.664137    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf379b9b0_301a_47af_8703_b07da27e797e.slice/crio-688ee7695ff08c0207e9166b72dc09cef5b4a842ed5bfcbfe8c38c7400244146 WatchSource:0}: Error finding container 688ee7695ff08c0207e9166b72dc09cef5b4a842ed5bfcbfe8c38c7400244146: Status 404 returned error can't find the container with id 688ee7695ff08c0207e9166b72dc09cef5b4a842ed5bfcbfe8c38c7400244146
Sep 09 20:12:30.716116 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.716092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdx6r" event={"ID":"f379b9b0-301a-47af-8703-b07da27e797e","Type":"ContainerStarted","Data":"688ee7695ff08c0207e9166b72dc09cef5b4a842ed5bfcbfe8c38c7400244146"}
Sep 09 20:12:30.717119 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:30.717103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qkfk2" event={"ID":"a80d1370-4cbe-4f65-8dd5-d42457f347b2","Type":"ContainerStarted","Data":"ca981e993b20d18d249a8556f612f396e8d7e1bab0041ca7c6b003c505c09bed"}
Sep 09 20:12:35.733022 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:35.732973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdx6r" event={"ID":"f379b9b0-301a-47af-8703-b07da27e797e","Type":"ContainerStarted","Data":"68955ff5e418a25aba7704d89929f25ef46d44ddb7832b0115627b79aae7c7cb"}
Sep 09 20:12:35.733514 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:35.733114    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:12:35.734524 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:35.734501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qkfk2" event={"ID":"a80d1370-4cbe-4f65-8dd5-d42457f347b2","Type":"ContainerStarted","Data":"1e47e013b9afa9ea7ad9823f19f4c1d5da113578c7c01eb553f9b44b19c76c53"}
Sep 09 20:12:35.750251 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:35.750208    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sdx6r" podStartSLOduration=67.225936218 podStartE2EDuration="1m11.750195794s" totalImagesPullingTime="4.524259576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:35.748584464 +0000 UTC m=+71.722354530" watchObservedRunningTime="2026-09-09 20:12:35.750195794 +0000 UTC m=+71.723965892"
Sep 09 20:12:35.763043 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:12:35.762985    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qkfk2" podStartSLOduration=67.232843525 podStartE2EDuration="1m11.762970887s" totalImagesPullingTime="4.530127362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:35.762761327 +0000 UTC m=+71.736531392" watchObservedRunningTime="2026-09-09 20:12:35.762970887 +0000 UTC m=+71.736740991"
Sep 09 20:13:01.855382 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:13:01.855280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:13:01.855781 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:13:01.855435    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:13:01.855781 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:13:01.855508    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:14:05.855493451 +0000 UTC m=+161.829263492 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:13:06.738596 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:13:06.738563    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sdx6r"
Sep 09 20:13:34.272787 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:13:34.272749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:13:34.273372 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:13:34.272936    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:13:34.273372 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:13:34.273068    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:36.273041914 +0000 UTC m=+252.246811975 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : secret "metrics-daemon-secret" not found
Sep 09 20:14:00.997380 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:14:00.997331    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jghvp" podUID="a6313d4f-b898-4da8-8010-c67a8726dfea"
Sep 09 20:14:01.768065 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.767976    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"]
Sep 09 20:14:01.770671 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.770654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:01.773037 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.773013    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:14:01.774340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.774141    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:14:01.774340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.774229    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dgck7\""
Sep 09 20:14:01.774340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.774307    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:14:01.774340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.774311    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 20:14:01.775183 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.775163    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"]
Sep 09 20:14:01.933938 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.933911    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jghvp"
Sep 09 20:14:01.952472 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.952448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/420c4d22-f5ed-44a7-97a1-31fa5daf5ac4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69b88874d4-xmgbw\" (UID: \"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:01.952554 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:01.952480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rfzk\" (UniqueName: \"kubernetes.io/projected/420c4d22-f5ed-44a7-97a1-31fa5daf5ac4-kube-api-access-6rfzk\") pod \"managed-serviceaccount-addon-agent-69b88874d4-xmgbw\" (UID: \"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:02.053874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:02.053806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/420c4d22-f5ed-44a7-97a1-31fa5daf5ac4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69b88874d4-xmgbw\" (UID: \"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:02.053874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:02.053847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rfzk\" (UniqueName: \"kubernetes.io/projected/420c4d22-f5ed-44a7-97a1-31fa5daf5ac4-kube-api-access-6rfzk\") pod \"managed-serviceaccount-addon-agent-69b88874d4-xmgbw\" (UID: \"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:02.056330 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:02.056309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/420c4d22-f5ed-44a7-97a1-31fa5daf5ac4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69b88874d4-xmgbw\" (UID: \"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:02.061780 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:02.061753    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rfzk\" (UniqueName: \"kubernetes.io/projected/420c4d22-f5ed-44a7-97a1-31fa5daf5ac4-kube-api-access-6rfzk\") pod \"managed-serviceaccount-addon-agent-69b88874d4-xmgbw\" (UID: \"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:02.089774 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:02.089747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"
Sep 09 20:14:02.355109 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:02.355032    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw"]
Sep 09 20:14:02.358325 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:14:02.358280    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod420c4d22_f5ed_44a7_97a1_31fa5daf5ac4.slice/crio-b91f1013c62029ddf1820a864f06186e3e3e64df578991c714b928c5bd8e958f WatchSource:0}: Error finding container b91f1013c62029ddf1820a864f06186e3e3e64df578991c714b928c5bd8e958f: Status 404 returned error can't find the container with id b91f1013c62029ddf1820a864f06186e3e3e64df578991c714b928c5bd8e958f
Sep 09 20:14:02.472072 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:14:02.472025    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:14:02.937170 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:02.937136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw" event={"ID":"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4","Type":"ContainerStarted","Data":"b91f1013c62029ddf1820a864f06186e3e3e64df578991c714b928c5bd8e958f"}
Sep 09 20:14:04.943805 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:04.943770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw" event={"ID":"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4","Type":"ContainerStarted","Data":"338467c61d5934b768183ce96d042dcd7c40abb0d949548f7ae41251056e1e66"}
Sep 09 20:14:04.958422 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:04.958371    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw" podStartSLOduration=1.493206271 podStartE2EDuration="3.958355287s" totalImagesPullingTime="2.465149016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:04.957819478 +0000 UTC m=+160.931589540" watchObservedRunningTime="2026-09-09 20:14:04.958355287 +0000 UTC m=+160.932125386"
Sep 09 20:14:05.882096 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:05.882057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:14:05.882266 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:14:05.882201    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:14:05.882309 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:14:05.882267    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls podName:a6313d4f-b898-4da8-8010-c67a8726dfea nodeName:}" failed. No retries permitted until 2026-09-09 20:16:07.882250827 +0000 UTC m=+283.856020897 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls") pod "dns-default-jghvp" (UID: "a6313d4f-b898-4da8-8010-c67a8726dfea") : secret "dns-default-metrics-tls" not found
Sep 09 20:14:14.460850 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:14:14.460820    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:15:36.282585 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:36.282531    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:15:36.283103 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:36.282683    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:15:36.283103 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:36.282764    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs podName:b355c14c-96e4-4d5d-ada5-6142cd99e951 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:38.282740684 +0000 UTC m=+374.256510726 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs") pod "network-metrics-daemon-zq8t9" (UID: "b355c14c-96e4-4d5d-ada5-6142cd99e951") : secret "metrics-daemon-secret" not found
Sep 09 20:15:39.770128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.770091    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-5h772"]
Sep 09 20:15:39.772969 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.772949    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-5h772"
Sep 09 20:15:39.774227 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.774207    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"]
Sep 09 20:15:39.776742 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.776727    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"
Sep 09 20:15:39.777336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.777312    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dgskb\""
Sep 09 20:15:39.780347 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.780316    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.780523 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.780499    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-blrtb\""
Sep 09 20:15:39.781483 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.781468    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.782199 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.782186    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-xchkg"]
Sep 09 20:15:39.784790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.784772    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-k99c5"]
Sep 09 20:15:39.784885 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.784870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:39.787945 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.787842    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:39.788451 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.788409    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-p8qmk\""
Sep 09 20:15:39.788733 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.788713    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 20:15:39.788820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.788720    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.788867 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.788729    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-5h772"]
Sep 09 20:15:39.789160 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.789140    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 20:15:39.789659 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.789636    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.794622 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.794600    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 20:15:39.794759 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.794731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.794973 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.794602    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.795202 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.795025    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 20:15:39.795401 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.794677    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m5tq6\""
Sep 09 20:15:39.795859 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.795837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"]
Sep 09 20:15:39.799186 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.799168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-xchkg"]
Sep 09 20:15:39.800685 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.800662    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-k99c5"]
Sep 09 20:15:39.800790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.800770    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 20:15:39.803580 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.803561    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 20:15:39.874711 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.874676    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"]
Sep 09 20:15:39.878017 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.877986    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"]
Sep 09 20:15:39.878143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.878105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:39.880597 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.880578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:39.882060 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.882042    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.882165 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.882143    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5prhj\""
Sep 09 20:15:39.882246 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.882154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 20:15:39.882246 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.882228    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.882458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.882442    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 20:15:39.883824 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.883798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 20:15:39.885253 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.885234    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.885411 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.885364    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5vcvh\""
Sep 09 20:15:39.885411 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.885380    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 20:15:39.885541 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.885443    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.894700 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.894675    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"]
Sep 09 20:15:39.895384 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.895363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"]
Sep 09 20:15:39.906031 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5fj\" (UniqueName: \"kubernetes.io/projected/e8ee55b2-06d4-4686-b4f4-a2e52559a2ee-kube-api-access-bz5fj\") pod \"volume-data-source-validator-695c895bd6-rj5j7\" (UID: \"e8ee55b2-06d4-4686-b4f4-a2e52559a2ee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"
Sep 09 20:15:39.906126 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-trusted-ca\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:39.906126 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/41ae5d00-e0ab-425b-bb62-7452b59c2101-snapshots\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:39.906126 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrw9\" (UniqueName: \"kubernetes.io/projected/1c9804fd-bdb0-47b7-a668-efdfab71a1d0-kube-api-access-slrw9\") pod \"network-check-source-685456655f-5h772\" (UID: \"1c9804fd-bdb0-47b7-a668-efdfab71a1d0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-5h772"
Sep 09 20:15:39.906126 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41ae5d00-e0ab-425b-bb62-7452b59c2101-serving-cert\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:39.906307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj68l\" (UniqueName: \"kubernetes.io/projected/41ae5d00-e0ab-425b-bb62-7452b59c2101-kube-api-access-xj68l\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:39.906307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ae5d00-e0ab-425b-bb62-7452b59c2101-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:39.906307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ae5d00-e0ab-425b-bb62-7452b59c2101-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:39.906307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvvq\" (UniqueName: \"kubernetes.io/projected/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-kube-api-access-wkvvq\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:39.906307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-config\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:39.906486 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-serving-cert\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:39.906486 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:39.906325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41ae5d00-e0ab-425b-bb62-7452b59c2101-tmp\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.007358 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5fj\" (UniqueName: \"kubernetes.io/projected/e8ee55b2-06d4-4686-b4f4-a2e52559a2ee-kube-api-access-bz5fj\") pod \"volume-data-source-validator-695c895bd6-rj5j7\" (UID: \"e8ee55b2-06d4-4686-b4f4-a2e52559a2ee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"
Sep 09 20:15:40.007358 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-trusted-ca\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.007610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/41ae5d00-e0ab-425b-bb62-7452b59c2101-snapshots\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.007610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.007610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrw9\" (UniqueName: \"kubernetes.io/projected/1c9804fd-bdb0-47b7-a668-efdfab71a1d0-kube-api-access-slrw9\") pod \"network-check-source-685456655f-5h772\" (UID: \"1c9804fd-bdb0-47b7-a668-efdfab71a1d0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-5h772"
Sep 09 20:15:40.007610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41ae5d00-e0ab-425b-bb62-7452b59c2101-serving-cert\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.007610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj68l\" (UniqueName: \"kubernetes.io/projected/41ae5d00-e0ab-425b-bb62-7452b59c2101-kube-api-access-xj68l\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.007610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ae5d00-e0ab-425b-bb62-7452b59c2101-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.007610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ae5d00-e0ab-425b-bb62-7452b59c2101-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1c687f-3564-4380-b568-7d619a225fe3-serving-cert\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvvq\" (UniqueName: \"kubernetes.io/projected/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-kube-api-access-wkvvq\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-config\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hqt\" (UniqueName: \"kubernetes.io/projected/2b1c687f-3564-4380-b568-7d619a225fe3-kube-api-access-v7hqt\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b1c687f-3564-4380-b568-7d619a225fe3-config\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-serving-cert\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.007966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41ae5d00-e0ab-425b-bb62-7452b59c2101-tmp\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.008374 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.007983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8w5\" (UniqueName: \"kubernetes.io/projected/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-kube-api-access-tz8w5\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.008374 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.008298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ae5d00-e0ab-425b-bb62-7452b59c2101-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.008374 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.008306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/41ae5d00-e0ab-425b-bb62-7452b59c2101-snapshots\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.008374 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.008372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-trusted-ca\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.008579 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.008568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41ae5d00-e0ab-425b-bb62-7452b59c2101-tmp\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.008631 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.008585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ae5d00-e0ab-425b-bb62-7452b59c2101-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.008702 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.008680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-config\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.009977 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.009958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41ae5d00-e0ab-425b-bb62-7452b59c2101-serving-cert\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.010199 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.010182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-serving-cert\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.016336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.016310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrw9\" (UniqueName: \"kubernetes.io/projected/1c9804fd-bdb0-47b7-a668-efdfab71a1d0-kube-api-access-slrw9\") pod \"network-check-source-685456655f-5h772\" (UID: \"1c9804fd-bdb0-47b7-a668-efdfab71a1d0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-5h772"
Sep 09 20:15:40.016460 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.016408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj68l\" (UniqueName: \"kubernetes.io/projected/41ae5d00-e0ab-425b-bb62-7452b59c2101-kube-api-access-xj68l\") pod \"insights-operator-76bcd6ccf5-xchkg\" (UID: \"41ae5d00-e0ab-425b-bb62-7452b59c2101\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.016534 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.016503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5fj\" (UniqueName: \"kubernetes.io/projected/e8ee55b2-06d4-4686-b4f4-a2e52559a2ee-kube-api-access-bz5fj\") pod \"volume-data-source-validator-695c895bd6-rj5j7\" (UID: \"e8ee55b2-06d4-4686-b4f4-a2e52559a2ee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"
Sep 09 20:15:40.016534 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.016512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvvq\" (UniqueName: \"kubernetes.io/projected/b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a-kube-api-access-wkvvq\") pod \"console-operator-85b454874b-k99c5\" (UID: \"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a\") " pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.083321 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.083247    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-5h772"
Sep 09 20:15:40.088362 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.088344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"
Sep 09 20:15:40.096862 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.096841    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg"
Sep 09 20:15:40.106914 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.106838    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:40.109172 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.109136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.109396 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.109198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1c687f-3564-4380-b568-7d619a225fe3-serving-cert\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.109396 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.109238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.109396 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:40.109279    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:40.109396 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.109294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hqt\" (UniqueName: \"kubernetes.io/projected/2b1c687f-3564-4380-b568-7d619a225fe3-kube-api-access-v7hqt\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.109396 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.109329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b1c687f-3564-4380-b568-7d619a225fe3-config\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.109396 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:40.109352    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls podName:f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:40.609329223 +0000 UTC m=+256.583099264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lhznr" (UID: "f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:40.109818 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.109721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8w5\" (UniqueName: \"kubernetes.io/projected/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-kube-api-access-tz8w5\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.110769 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.110285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b1c687f-3564-4380-b568-7d619a225fe3-config\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.110769 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.110519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.112251 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.112214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1c687f-3564-4380-b568-7d619a225fe3-serving-cert\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.117737 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.117714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8w5\" (UniqueName: \"kubernetes.io/projected/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-kube-api-access-tz8w5\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.118964 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.118937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hqt\" (UniqueName: \"kubernetes.io/projected/2b1c687f-3564-4380-b568-7d619a225fe3-kube-api-access-v7hqt\") pod \"service-ca-operator-859f94c4b-88f8z\" (UID: \"2b1c687f-3564-4380-b568-7d619a225fe3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.188439 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.188405    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"
Sep 09 20:15:40.360578 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.360506    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-5h772"]
Sep 09 20:15:40.363276 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.363250    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7"]
Sep 09 20:15:40.363593 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:15:40.363570    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9804fd_bdb0_47b7_a668_efdfab71a1d0.slice/crio-65387d562d965cdc4844f1354022da6fec881e576aa1a5bd9e283d9f13c96da6 WatchSource:0}: Error finding container 65387d562d965cdc4844f1354022da6fec881e576aa1a5bd9e283d9f13c96da6: Status 404 returned error can't find the container with id 65387d562d965cdc4844f1354022da6fec881e576aa1a5bd9e283d9f13c96da6
Sep 09 20:15:40.366158 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:15:40.366134    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ee55b2_06d4_4686_b4f4_a2e52559a2ee.slice/crio-412c74bf42fd9bd42a2943134b1b196f1ec8240b646226f11932933f2113ae7e WatchSource:0}: Error finding container 412c74bf42fd9bd42a2943134b1b196f1ec8240b646226f11932933f2113ae7e: Status 404 returned error can't find the container with id 412c74bf42fd9bd42a2943134b1b196f1ec8240b646226f11932933f2113ae7e
Sep 09 20:15:40.394380 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.394354    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-k99c5"]
Sep 09 20:15:40.396740 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.396716    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-xchkg"]
Sep 09 20:15:40.397292 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:15:40.397266    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ceae69_6511_4bd5_ba0e_c4a2fbbbb38a.slice/crio-4bc8a436f566676d261901a6e1eb36281268fa95dbbb091bf3a8fd5f6e47f7ec WatchSource:0}: Error finding container 4bc8a436f566676d261901a6e1eb36281268fa95dbbb091bf3a8fd5f6e47f7ec: Status 404 returned error can't find the container with id 4bc8a436f566676d261901a6e1eb36281268fa95dbbb091bf3a8fd5f6e47f7ec
Sep 09 20:15:40.399874 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:15:40.399816    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ae5d00_e0ab_425b_bb62_7452b59c2101.slice/crio-c5417a796bcdf1bef028983ac881a1691ac4d2fd23415409c4b37d73ac25411c WatchSource:0}: Error finding container c5417a796bcdf1bef028983ac881a1691ac4d2fd23415409c4b37d73ac25411c: Status 404 returned error can't find the container with id c5417a796bcdf1bef028983ac881a1691ac4d2fd23415409c4b37d73ac25411c
Sep 09 20:15:40.455932 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.455901    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z"]
Sep 09 20:15:40.459151 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:15:40.459122    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1c687f_3564_4380_b568_7d619a225fe3.slice/crio-a0f6bc1f536651bfac52e9ffb77b4358f7a683a9ef5f2dc1a998715ab32560e0 WatchSource:0}: Error finding container a0f6bc1f536651bfac52e9ffb77b4358f7a683a9ef5f2dc1a998715ab32560e0: Status 404 returned error can't find the container with id a0f6bc1f536651bfac52e9ffb77b4358f7a683a9ef5f2dc1a998715ab32560e0
Sep 09 20:15:40.614033 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:40.613932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:40.614203 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:40.614144    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:40.614267 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:40.614235    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls podName:f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:41.614214109 +0000 UTC m=+257.587984170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lhznr" (UID: "f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:41.179863 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.179748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z" event={"ID":"2b1c687f-3564-4380-b568-7d619a225fe3","Type":"ContainerStarted","Data":"a0f6bc1f536651bfac52e9ffb77b4358f7a683a9ef5f2dc1a998715ab32560e0"}
Sep 09 20:15:41.182012 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.181942    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-k99c5" event={"ID":"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a","Type":"ContainerStarted","Data":"4bc8a436f566676d261901a6e1eb36281268fa95dbbb091bf3a8fd5f6e47f7ec"}
Sep 09 20:15:41.183334 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.183310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg" event={"ID":"41ae5d00-e0ab-425b-bb62-7452b59c2101","Type":"ContainerStarted","Data":"c5417a796bcdf1bef028983ac881a1691ac4d2fd23415409c4b37d73ac25411c"}
Sep 09 20:15:41.184843 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.184819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7" event={"ID":"e8ee55b2-06d4-4686-b4f4-a2e52559a2ee","Type":"ContainerStarted","Data":"412c74bf42fd9bd42a2943134b1b196f1ec8240b646226f11932933f2113ae7e"}
Sep 09 20:15:41.187606 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.187585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-5h772" event={"ID":"1c9804fd-bdb0-47b7-a668-efdfab71a1d0","Type":"ContainerStarted","Data":"d6482601d42d7e5ebe025f5e4c4f77ed6c0140e4ef3ab8cda1e7bfff91139d5c"}
Sep 09 20:15:41.187708 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.187625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-5h772" event={"ID":"1c9804fd-bdb0-47b7-a668-efdfab71a1d0","Type":"ContainerStarted","Data":"65387d562d965cdc4844f1354022da6fec881e576aa1a5bd9e283d9f13c96da6"}
Sep 09 20:15:41.203886 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.203835    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-5h772" podStartSLOduration=2.203819293 podStartE2EDuration="2.203819293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:41.203388793 +0000 UTC m=+257.177158856" watchObservedRunningTime="2026-09-09 20:15:41.203819293 +0000 UTC m=+257.177589358"
Sep 09 20:15:41.628746 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:41.628155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:41.628746 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:41.628338    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:41.628746 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:41.628402    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls podName:f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:43.628383601 +0000 UTC m=+259.602153647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lhznr" (UID: "f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:42.194685 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:42.194597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7" event={"ID":"e8ee55b2-06d4-4686-b4f4-a2e52559a2ee","Type":"ContainerStarted","Data":"68eb391e5e2ac9b629a03e89c6b62ee7204ff0491bf2387cb4abdb228699496a"}
Sep 09 20:15:42.210931 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:42.210869    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rj5j7" podStartSLOduration=1.652094984 podStartE2EDuration="3.210846883s" totalImagesPullingTime="1.558751899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:42.209557734 +0000 UTC m=+258.183327798" watchObservedRunningTime="2026-09-09 20:15:42.210846883 +0000 UTC m=+258.184616946"
Sep 09 20:15:43.646652 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:43.646611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:43.647085 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:43.646731    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:43.647085 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:43.646815    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls podName:f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.646798818 +0000 UTC m=+263.620568882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lhznr" (UID: "f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:43.974597 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:43.974562    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"]
Sep 09 20:15:43.977910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:43.977893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"
Sep 09 20:15:43.981045 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:43.980747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 20:15:43.981045 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:43.980769    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9bqmx\""
Sep 09 20:15:43.981045 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:43.980810    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 20:15:43.987272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:43.987251    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"]
Sep 09 20:15:44.151383 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.151289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczcg\" (UniqueName: \"kubernetes.io/projected/48c041b3-39ae-40e7-b31a-dde9da515a76-kube-api-access-xczcg\") pod \"migrator-fdbb8fdff-k6d79\" (UID: \"48c041b3-39ae-40e7-b31a-dde9da515a76\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"
Sep 09 20:15:44.209066 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.209038    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/0.log"
Sep 09 20:15:44.209209 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.209079    2562 generic.go:356] "Generic (PLEG): container finished" podID="b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a" containerID="0c08407dd8511203b18b2e2a818a9a9839ca9d4afa51b82b411a35b2ec27ce17" exitCode=255
Sep 09 20:15:44.209209 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.209165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-k99c5" event={"ID":"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a","Type":"ContainerDied","Data":"0c08407dd8511203b18b2e2a818a9a9839ca9d4afa51b82b411a35b2ec27ce17"}
Sep 09 20:15:44.209451 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.209433    2562 scope.go:117] "RemoveContainer" containerID="0c08407dd8511203b18b2e2a818a9a9839ca9d4afa51b82b411a35b2ec27ce17"
Sep 09 20:15:44.211236 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.211210    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg" event={"ID":"41ae5d00-e0ab-425b-bb62-7452b59c2101","Type":"ContainerStarted","Data":"9ebb99e7bfeeb90d01165104d28c84da49ba1bf1cd81bb5ad3736886f0eb3c98"}
Sep 09 20:15:44.212960 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.212935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z" event={"ID":"2b1c687f-3564-4380-b568-7d619a225fe3","Type":"ContainerStarted","Data":"c6558be5c04dd49058194d5fc7efc9a5254c200ab666ab382e5a8f354519d428"}
Sep 09 20:15:44.244059 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.243991    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg" podStartSLOduration=1.7912809269999999 podStartE2EDuration="5.243973981s" totalImagesPullingTime="3.452693054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:44.242687824 +0000 UTC m=+260.216457888" watchObservedRunningTime="2026-09-09 20:15:44.243973981 +0000 UTC m=+260.217744045"
Sep 09 20:15:44.251680 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.251657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xczcg\" (UniqueName: \"kubernetes.io/projected/48c041b3-39ae-40e7-b31a-dde9da515a76-kube-api-access-xczcg\") pod \"migrator-fdbb8fdff-k6d79\" (UID: \"48c041b3-39ae-40e7-b31a-dde9da515a76\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"
Sep 09 20:15:44.260370 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.260315    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z" podStartSLOduration=1.865107278 podStartE2EDuration="5.260298458s" totalImagesPullingTime="3.39519118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:44.258876526 +0000 UTC m=+260.232646594" watchObservedRunningTime="2026-09-09 20:15:44.260298458 +0000 UTC m=+260.234068522"
Sep 09 20:15:44.266566 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.266544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczcg\" (UniqueName: \"kubernetes.io/projected/48c041b3-39ae-40e7-b31a-dde9da515a76-kube-api-access-xczcg\") pod \"migrator-fdbb8fdff-k6d79\" (UID: \"48c041b3-39ae-40e7-b31a-dde9da515a76\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"
Sep 09 20:15:44.352180 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.352155    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"
Sep 09 20:15:44.617854 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:44.617776    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79"]
Sep 09 20:15:44.620799 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:15:44.620773    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c041b3_39ae_40e7_b31a_dde9da515a76.slice/crio-1efa95f7d704fdb1aa9c710beba9f7a8e21c2e3d21817ae8875098227ca56dd4 WatchSource:0}: Error finding container 1efa95f7d704fdb1aa9c710beba9f7a8e21c2e3d21817ae8875098227ca56dd4: Status 404 returned error can't find the container with id 1efa95f7d704fdb1aa9c710beba9f7a8e21c2e3d21817ae8875098227ca56dd4
Sep 09 20:15:45.217803 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.217778    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:15:45.218235 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.218161    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/0.log"
Sep 09 20:15:45.218235 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.218194    2562 generic.go:356] "Generic (PLEG): container finished" podID="b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a" containerID="3eb56bb7127592abc2c7ebf74cba8b26ddff96a83eedbd140a5ad881cc6158af" exitCode=255
Sep 09 20:15:45.218343 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.218254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-k99c5" event={"ID":"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a","Type":"ContainerDied","Data":"3eb56bb7127592abc2c7ebf74cba8b26ddff96a83eedbd140a5ad881cc6158af"}
Sep 09 20:15:45.218343 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.218291    2562 scope.go:117] "RemoveContainer" containerID="0c08407dd8511203b18b2e2a818a9a9839ca9d4afa51b82b411a35b2ec27ce17"
Sep 09 20:15:45.218490 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.218470    2562 scope.go:117] "RemoveContainer" containerID="3eb56bb7127592abc2c7ebf74cba8b26ddff96a83eedbd140a5ad881cc6158af"
Sep 09 20:15:45.218669 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:45.218650    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-k99c5_openshift-console-operator(b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a)\"" pod="openshift-console-operator/console-operator-85b454874b-k99c5" podUID="b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a"
Sep 09 20:15:45.219570 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.219545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79" event={"ID":"48c041b3-39ae-40e7-b31a-dde9da515a76","Type":"ContainerStarted","Data":"1efa95f7d704fdb1aa9c710beba9f7a8e21c2e3d21817ae8875098227ca56dd4"}
Sep 09 20:15:45.575284 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.574942    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pdkmv"]
Sep 09 20:15:45.578634 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.578605    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.581163 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.581139    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m6kgz\""
Sep 09 20:15:45.581163 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.581153    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:15:45.581339 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.581256    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:15:45.587277 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.587034    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pdkmv"]
Sep 09 20:15:45.661814 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.661783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51288a2a-3ab9-4f14-9930-a0105a768158-crio-socket\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.661945 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.661911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6csh4\" (UniqueName: \"kubernetes.io/projected/51288a2a-3ab9-4f14-9930-a0105a768158-kube-api-access-6csh4\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.662025 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.661954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.662025 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.661994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51288a2a-3ab9-4f14-9930-a0105a768158-data-volume\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.662117 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.662078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.763012 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.762960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6csh4\" (UniqueName: \"kubernetes.io/projected/51288a2a-3ab9-4f14-9930-a0105a768158-kube-api-access-6csh4\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.763159 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.763024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.763159 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.763055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51288a2a-3ab9-4f14-9930-a0105a768158-data-volume\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.763159 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.763093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.763317 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.763160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51288a2a-3ab9-4f14-9930-a0105a768158-crio-socket\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.763317 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:45.763170    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:45.763317 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:45.763244    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:46.26322488 +0000 UTC m=+262.236994924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:45.763317 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:45.763243    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:45.763317 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.763258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51288a2a-3ab9-4f14-9930-a0105a768158-crio-socket\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.763317 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:45.763286    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:46.263276458 +0000 UTC m=+262.237046519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:45.763493 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.763348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51288a2a-3ab9-4f14-9930-a0105a768158-data-volume\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:45.772393 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:45.772371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6csh4\" (UniqueName: \"kubernetes.io/projected/51288a2a-3ab9-4f14-9930-a0105a768158-kube-api-access-6csh4\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:46.224283 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:46.224259    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:15:46.224591 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:46.224573    2562 scope.go:117] "RemoveContainer" containerID="3eb56bb7127592abc2c7ebf74cba8b26ddff96a83eedbd140a5ad881cc6158af"
Sep 09 20:15:46.224736 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:46.224715    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-k99c5_openshift-console-operator(b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a)\"" pod="openshift-console-operator/console-operator-85b454874b-k99c5" podUID="b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a"
Sep 09 20:15:46.226053 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:46.226031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79" event={"ID":"48c041b3-39ae-40e7-b31a-dde9da515a76","Type":"ContainerStarted","Data":"8ce3f0c3f8f9f2a33f9dfd1a958d687651ed0e117ce1f85fd9f6f215cd407a37"}
Sep 09 20:15:46.226127 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:46.226098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79" event={"ID":"48c041b3-39ae-40e7-b31a-dde9da515a76","Type":"ContainerStarted","Data":"1cc7d813e4a197524202f0562045b44a18c0d8da97b34eb58469d8c1830a16de"}
Sep 09 20:15:46.253876 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:46.253837    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-k6d79" podStartSLOduration=1.78658529 podStartE2EDuration="3.253824207s" totalImagesPullingTime="1.467238917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:46.253358918 +0000 UTC m=+262.227128982" watchObservedRunningTime="2026-09-09 20:15:46.253824207 +0000 UTC m=+262.227594302"
Sep 09 20:15:46.266846 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:46.266789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:46.266938 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:46.266923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:46.267021 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:46.266923    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:46.267021 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:46.266997    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.266980815 +0000 UTC m=+263.240750856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:46.267127 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:46.267018    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:46.267127 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:46.267073    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.267055914 +0000 UTC m=+263.240825960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:47.274918 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:47.274843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:47.274918 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:47.274886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:47.275311 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:47.274959    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:47.275311 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:47.274987    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:47.275311 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:47.275034    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:49.275017908 +0000 UTC m=+265.248787971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:47.275311 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:47.275058    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:49.275041264 +0000 UTC m=+265.248811311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:47.677443 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:47.677407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:47.677588 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:47.677516    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:47.677588 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:47.677568    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls podName:f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:55.677554272 +0000 UTC m=+271.651324313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lhznr" (UID: "f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:48.136783 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:48.136759    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2cdv6_cea6223c-9ead-467f-b0e8-c09ceb9ec51a/dns-node-resolver/0.log"
Sep 09 20:15:49.288844 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:49.288813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:49.289210 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:49.288857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:49.289210 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:49.288946    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:49.289210 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:49.288964    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:49.289210 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:49.289026    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:53.288991453 +0000 UTC m=+269.262761510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:49.289210 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:49.289043    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:53.289034846 +0000 UTC m=+269.262804886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:49.535677 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:49.535650    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6pfl_512ceb34-1d30-47d0-b1f4-e2a0871e519b/node-ca/0.log"
Sep 09 20:15:50.107567 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:50.107535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:50.107731 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:50.107590    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:15:50.107929 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:50.107914    2562 scope.go:117] "RemoveContainer" containerID="3eb56bb7127592abc2c7ebf74cba8b26ddff96a83eedbd140a5ad881cc6158af"
Sep 09 20:15:50.108155 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:50.108131    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-k99c5_openshift-console-operator(b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a)\"" pod="openshift-console-operator/console-operator-85b454874b-k99c5" podUID="b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a"
Sep 09 20:15:50.336022 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:50.335979    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s5bqc_66c4d70d-5928-4cc4-82c6-3179c6999aa7/serve-healthcheck-canary/0.log"
Sep 09 20:15:50.538459 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:50.538425    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-k6d79_48c041b3-39ae-40e7-b31a-dde9da515a76/migrator/0.log"
Sep 09 20:15:50.736284 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:50.736257    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-k6d79_48c041b3-39ae-40e7-b31a-dde9da515a76/graceful-termination/0.log"
Sep 09 20:15:53.321193 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:53.321157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:53.321617 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:53.321208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:15:53.321617 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:53.321316    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:53.321617 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:53.321327    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:53.321617 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:53.321410    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:01.321393165 +0000 UTC m=+277.295163222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:53.321617 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:53.321423    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:01.321416942 +0000 UTC m=+277.295186983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:55.741412 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:15:55.741372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:15:55.741788 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:55.741542    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:15:55.741788 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:15:55.741612    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls podName:f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:11.741594682 +0000 UTC m=+287.715364728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lhznr" (UID: "f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:01.379496 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:01.379453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:16:01.379888 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:01.379506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:16:01.379888 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:01.379597    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:16:01.379888 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:01.379661    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls podName:51288a2a-3ab9-4f14-9930-a0105a768158 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:17.379646847 +0000 UTC m=+293.353416892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pdkmv" (UID: "51288a2a-3ab9-4f14-9930-a0105a768158") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:16:01.380076 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:01.379982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51288a2a-3ab9-4f14-9930-a0105a768158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:16:04.461392 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:04.461361    2562 scope.go:117] "RemoveContainer" containerID="3eb56bb7127592abc2c7ebf74cba8b26ddff96a83eedbd140a5ad881cc6158af"
Sep 09 20:16:04.935422 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:04.935380    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jghvp" podUID="a6313d4f-b898-4da8-8010-c67a8726dfea"
Sep 09 20:16:05.285975 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.285887    2562 generic.go:356] "Generic (PLEG): container finished" podID="420c4d22-f5ed-44a7-97a1-31fa5daf5ac4" containerID="338467c61d5934b768183ce96d042dcd7c40abb0d949548f7ae41251056e1e66" exitCode=255
Sep 09 20:16:05.285975 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.285962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw" event={"ID":"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4","Type":"ContainerDied","Data":"338467c61d5934b768183ce96d042dcd7c40abb0d949548f7ae41251056e1e66"}
Sep 09 20:16:05.287903 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.287880    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:16:05.288043 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.287961    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jghvp"
Sep 09 20:16:05.288043 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.287992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-k99c5" event={"ID":"b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a","Type":"ContainerStarted","Data":"d0295781b80c0e21ca5dbd44d59510a488ac29fadae3318fa9af2b54ffaf6684"}
Sep 09 20:16:05.288354 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.288337    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:16:05.292314 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.292294    2562 scope.go:117] "RemoveContainer" containerID="338467c61d5934b768183ce96d042dcd7c40abb0d949548f7ae41251056e1e66"
Sep 09 20:16:05.315800 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.315750    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-k99c5" podStartSLOduration=22.865748559 podStartE2EDuration="26.315729829s" totalImagesPullingTime="3.44998127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:05.315188441 +0000 UTC m=+281.288958497" watchObservedRunningTime="2026-09-09 20:16:05.315729829 +0000 UTC m=+281.289499894"
Sep 09 20:16:05.905651 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:05.905621    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-k99c5"
Sep 09 20:16:06.292604 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.292518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69b88874d4-xmgbw" event={"ID":"420c4d22-f5ed-44a7-97a1-31fa5daf5ac4","Type":"ContainerStarted","Data":"8fabb70e22e1e755e26d51e944cee524da0379e65f35868e4a91a1c5dbe1dd0c"}
Sep 09 20:16:06.564045 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.563943    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d957f96b-g54nh"]
Sep 09 20:16:06.579137 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.579107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.583378 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.583355    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:16:06.583524 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.583416    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:16:06.583795 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.583781    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:16:06.583967 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.583947    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ctkpk\""
Sep 09 20:16:06.595994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.592714    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:16:06.604530 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.604504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d957f96b-g54nh"]
Sep 09 20:16:06.616818 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.616789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-image-registry-private-configuration\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.616971 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.616834    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-registry-certificates\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.616971 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.616855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-trusted-ca\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.616971 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.616872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-registry-tls\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.616971 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.616956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-ca-trust-extracted\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.617154 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.617031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8q4n\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-kube-api-access-k8q4n\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.617154 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.617077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-bound-sa-token\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.617154 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.617096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-installation-pull-secrets\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718252 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-image-registry-private-configuration\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718252 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-registry-certificates\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718483 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-trusted-ca\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718483 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-registry-tls\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718483 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-ca-trust-extracted\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718636 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8q4n\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-kube-api-access-k8q4n\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718636 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-bound-sa-token\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718636 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-installation-pull-secrets\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.718932 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.718902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-ca-trust-extracted\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.719167 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.719142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-registry-certificates\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.719484 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.719444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-trusted-ca\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.720783 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.720765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-image-registry-private-configuration\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.721382 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.721360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-registry-tls\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.721470 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.721389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-installation-pull-secrets\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.730208 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.730183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-bound-sa-token\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.730353 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.730335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8q4n\" (UniqueName: \"kubernetes.io/projected/d21b8b49-a6b0-4b27-9805-e7030cbd72bd-kube-api-access-k8q4n\") pod \"image-registry-7d957f96b-g54nh\" (UID: \"d21b8b49-a6b0-4b27-9805-e7030cbd72bd\") " pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:06.888440 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:06.888403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:07.186394 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:07.186310    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d957f96b-g54nh"]
Sep 09 20:16:07.189364 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:07.189339    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21b8b49_a6b0_4b27_9805_e7030cbd72bd.slice/crio-fac8fafdb7ea8ce44b16cddcc104df5ecf878f04fb6fc9a74860fcd586f34443 WatchSource:0}: Error finding container fac8fafdb7ea8ce44b16cddcc104df5ecf878f04fb6fc9a74860fcd586f34443: Status 404 returned error can't find the container with id fac8fafdb7ea8ce44b16cddcc104df5ecf878f04fb6fc9a74860fcd586f34443
Sep 09 20:16:07.298136 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:07.298105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d957f96b-g54nh" event={"ID":"d21b8b49-a6b0-4b27-9805-e7030cbd72bd","Type":"ContainerStarted","Data":"abc4b4d0b13faf2354272b0b6a8dc1a72254c8cc8b18b17b11c429ed3b56073b"}
Sep 09 20:16:07.298250 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:07.298156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d957f96b-g54nh" event={"ID":"d21b8b49-a6b0-4b27-9805-e7030cbd72bd","Type":"ContainerStarted","Data":"fac8fafdb7ea8ce44b16cddcc104df5ecf878f04fb6fc9a74860fcd586f34443"}
Sep 09 20:16:07.929458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:07.929418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:16:07.931808 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:07.931782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6313d4f-b898-4da8-8010-c67a8726dfea-metrics-tls\") pod \"dns-default-jghvp\" (UID: \"a6313d4f-b898-4da8-8010-c67a8726dfea\") " pod="openshift-dns/dns-default-jghvp"
Sep 09 20:16:07.990840 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:07.990802    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nht2s\""
Sep 09 20:16:07.998910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:07.998875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jghvp"
Sep 09 20:16:08.301066 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:08.300979    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:08.303061 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:08.303034    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jghvp"]
Sep 09 20:16:08.306294 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:08.306273    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6313d4f_b898_4da8_8010_c67a8726dfea.slice/crio-d2e09be2de1677f4a94108ed15aeabbf86c2e6cf2ec5b15f8fb7377e98dc2d15 WatchSource:0}: Error finding container d2e09be2de1677f4a94108ed15aeabbf86c2e6cf2ec5b15f8fb7377e98dc2d15: Status 404 returned error can't find the container with id d2e09be2de1677f4a94108ed15aeabbf86c2e6cf2ec5b15f8fb7377e98dc2d15
Sep 09 20:16:08.319352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:08.319302    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d957f96b-g54nh" podStartSLOduration=2.319286664 podStartE2EDuration="2.319286664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:08.317983854 +0000 UTC m=+284.291753919" watchObservedRunningTime="2026-09-09 20:16:08.319286664 +0000 UTC m=+284.293056737"
Sep 09 20:16:09.305194 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:09.305144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jghvp" event={"ID":"a6313d4f-b898-4da8-8010-c67a8726dfea","Type":"ContainerStarted","Data":"d2e09be2de1677f4a94108ed15aeabbf86c2e6cf2ec5b15f8fb7377e98dc2d15"}
Sep 09 20:16:10.310702 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:10.310669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jghvp" event={"ID":"a6313d4f-b898-4da8-8010-c67a8726dfea","Type":"ContainerStarted","Data":"4e2a459978bc76557acc2915d77612e9d30b399e3cae06ffdd1ee9a6f9756697"}
Sep 09 20:16:10.311077 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:10.310733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jghvp" event={"ID":"a6313d4f-b898-4da8-8010-c67a8726dfea","Type":"ContainerStarted","Data":"5c624e47a67243c1fdc026d1588649119b0d31cc3b6507465ab75f94394d6aba"}
Sep 09 20:16:10.311077 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:10.310770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jghvp"
Sep 09 20:16:10.326605 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:10.326527    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jghvp" podStartSLOduration=251.511357987 podStartE2EDuration="4m13.326510668s" totalImagesPullingTime="1.815152681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:10.325598862 +0000 UTC m=+286.299368924" watchObservedRunningTime="2026-09-09 20:16:10.326510668 +0000 UTC m=+286.300280723"
Sep 09 20:16:11.762342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:11.762298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:16:11.764615 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:11.764586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lhznr\" (UID: \"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:16:11.992686 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:11.992646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"
Sep 09 20:16:12.259966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:12.259932    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr"]
Sep 09 20:16:12.262774 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:12.262738    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9482af9_54ad_4ac9_8b2d_3c1ad7c16ee0.slice/crio-baf00d002165bae487d978d39adb1b5a8aaa233d6e13cc895486b93529bafede WatchSource:0}: Error finding container baf00d002165bae487d978d39adb1b5a8aaa233d6e13cc895486b93529bafede: Status 404 returned error can't find the container with id baf00d002165bae487d978d39adb1b5a8aaa233d6e13cc895486b93529bafede
Sep 09 20:16:12.321625 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:12.321540    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr" event={"ID":"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0","Type":"ContainerStarted","Data":"baf00d002165bae487d978d39adb1b5a8aaa233d6e13cc895486b93529bafede"}
Sep 09 20:16:15.135414 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.135381    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"]
Sep 09 20:16:15.137432 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.137415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"
Sep 09 20:16:15.139824 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.139799    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 20:16:15.139925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.139872    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6mgvq\""
Sep 09 20:16:15.146049 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.146026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"]
Sep 09 20:16:15.193319 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.193284    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58487adc-4fd6-46c1-871c-974bad59f921-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-rx2zk\" (UID: \"58487adc-4fd6-46c1-871c-974bad59f921\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"
Sep 09 20:16:15.294474 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.294441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58487adc-4fd6-46c1-871c-974bad59f921-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-rx2zk\" (UID: \"58487adc-4fd6-46c1-871c-974bad59f921\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"
Sep 09 20:16:15.296916 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.296882    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58487adc-4fd6-46c1-871c-974bad59f921-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-rx2zk\" (UID: \"58487adc-4fd6-46c1-871c-974bad59f921\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"
Sep 09 20:16:15.334160 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.334128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr" event={"ID":"f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0","Type":"ContainerStarted","Data":"0171ea4609b3ea8fb98b811f62c6ed45dc0d405d6f0ee2f20391758e3f90d30a"}
Sep 09 20:16:15.354152 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.354096    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lhznr" podStartSLOduration=33.94642352 podStartE2EDuration="36.354074374s" totalImagesPullingTime="2.407650854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:15.353276666 +0000 UTC m=+291.327046719" watchObservedRunningTime="2026-09-09 20:16:15.354074374 +0000 UTC m=+291.327844438"
Sep 09 20:16:15.447144 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.447058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"
Sep 09 20:16:15.713603 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:15.713531    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"]
Sep 09 20:16:15.716419 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:15.716393    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58487adc_4fd6_46c1_871c_974bad59f921.slice/crio-b2addb4df4de55ea08be29ddbc122949b0c5a39e2f07796d50b25ed1a018734b WatchSource:0}: Error finding container b2addb4df4de55ea08be29ddbc122949b0c5a39e2f07796d50b25ed1a018734b: Status 404 returned error can't find the container with id b2addb4df4de55ea08be29ddbc122949b0c5a39e2f07796d50b25ed1a018734b
Sep 09 20:16:16.338815 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:16.338780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk" event={"ID":"58487adc-4fd6-46c1-871c-974bad59f921","Type":"ContainerStarted","Data":"b2addb4df4de55ea08be29ddbc122949b0c5a39e2f07796d50b25ed1a018734b"}
Sep 09 20:16:17.344994 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.344900    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk" event={"ID":"58487adc-4fd6-46c1-871c-974bad59f921","Type":"ContainerStarted","Data":"b6e414d6005ae631195c783736aa627154739e96a58cb1f22849c743271ef89b"}
Sep 09 20:16:17.345424 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.345137    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"
Sep 09 20:16:17.350100 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.350079    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk"
Sep 09 20:16:17.361619 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.361574    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-rx2zk" podStartSLOduration=1.065419827 podStartE2EDuration="2.361560268s" totalImagesPullingTime="1.296140441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:17.360963748 +0000 UTC m=+293.334733810" watchObservedRunningTime="2026-09-09 20:16:17.361560268 +0000 UTC m=+293.335330333"
Sep 09 20:16:17.410699 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.410664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:16:17.412880 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.412853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51288a2a-3ab9-4f14-9930-a0105a768158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdkmv\" (UID: \"51288a2a-3ab9-4f14-9930-a0105a768158\") " pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:16:17.462109 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:17.462049    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zq8t9" podUID="b355c14c-96e4-4d5d-ada5-6142cd99e951"
Sep 09 20:16:17.690404 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.690366    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pdkmv"
Sep 09 20:16:17.959564 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:17.959539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pdkmv"]
Sep 09 20:16:17.961872 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:17.961848    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51288a2a_3ab9_4f14_9930_a0105a768158.slice/crio-0fe4a619dee0bc2c0b4cc21bf83a2f819f6800d9dfdc102f302774811f7a43ca WatchSource:0}: Error finding container 0fe4a619dee0bc2c0b4cc21bf83a2f819f6800d9dfdc102f302774811f7a43ca: Status 404 returned error can't find the container with id 0fe4a619dee0bc2c0b4cc21bf83a2f819f6800d9dfdc102f302774811f7a43ca
Sep 09 20:16:18.197431 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.197397    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"]
Sep 09 20:16:18.199409 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.199391    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.202134 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.202090    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 20:16:18.202134 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.202099    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-n87n8\""
Sep 09 20:16:18.202134 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.202108    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:16:18.202314 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.202098    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 20:16:18.215989 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.215934    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"]
Sep 09 20:16:18.316696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.316660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d0d5-3962-4ae8-833d-e025a5fc532b-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.316696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.316702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6gl\" (UniqueName: \"kubernetes.io/projected/bda2d0d5-3962-4ae8-833d-e025a5fc532b-kube-api-access-hn6gl\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.316923 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.316739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.316923 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.316794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.350073 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.350040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdkmv" event={"ID":"51288a2a-3ab9-4f14-9930-a0105a768158","Type":"ContainerStarted","Data":"ca0b0ab96370f2d98d1884325a6f79f1d5f8c2f1ea5c004f6a9e9355bc58e53b"}
Sep 09 20:16:18.350412 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.350094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdkmv" event={"ID":"51288a2a-3ab9-4f14-9930-a0105a768158","Type":"ContainerStarted","Data":"0fe4a619dee0bc2c0b4cc21bf83a2f819f6800d9dfdc102f302774811f7a43ca"}
Sep 09 20:16:18.417777 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.417744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d0d5-3962-4ae8-833d-e025a5fc532b-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.417966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.417784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6gl\" (UniqueName: \"kubernetes.io/projected/bda2d0d5-3962-4ae8-833d-e025a5fc532b-kube-api-access-hn6gl\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.417966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.417816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.417966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.417851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.418166 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:18.417990    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 20:16:18.418166 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:18.418082    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-tls podName:bda2d0d5-3962-4ae8-833d-e025a5fc532b nodeName:}" failed. No retries permitted until 2026-09-09 20:16:18.918060898 +0000 UTC m=+294.891830949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-hx8mr" (UID: "bda2d0d5-3962-4ae8-833d-e025a5fc532b") : secret "prometheus-operator-tls" not found
Sep 09 20:16:18.418466 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.418438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d0d5-3962-4ae8-833d-e025a5fc532b-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.420258 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.420233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.428412 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.428387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6gl\" (UniqueName: \"kubernetes.io/projected/bda2d0d5-3962-4ae8-833d-e025a5fc532b-kube-api-access-hn6gl\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.922612 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.922558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:18.924937 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:18.924907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda2d0d5-3962-4ae8-833d-e025a5fc532b-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-hx8mr\" (UID: \"bda2d0d5-3962-4ae8-833d-e025a5fc532b\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:19.108043 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:19.108021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"
Sep 09 20:16:19.357780 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:19.357741    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdkmv" event={"ID":"51288a2a-3ab9-4f14-9930-a0105a768158","Type":"ContainerStarted","Data":"b6737aea316cbbe0234e4146df6513b019cec77a12a7c5c0f20e67e67fbc26f8"}
Sep 09 20:16:19.375700 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:19.375674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-hx8mr"]
Sep 09 20:16:19.378151 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:19.378128    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda2d0d5_3962_4ae8_833d_e025a5fc532b.slice/crio-33b4d1e9ba7dcda7067af3801f280761efff7104ebb621bdd620f34cf7c5ba22 WatchSource:0}: Error finding container 33b4d1e9ba7dcda7067af3801f280761efff7104ebb621bdd620f34cf7c5ba22: Status 404 returned error can't find the container with id 33b4d1e9ba7dcda7067af3801f280761efff7104ebb621bdd620f34cf7c5ba22
Sep 09 20:16:20.320336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:20.320308    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jghvp"
Sep 09 20:16:20.363873 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:20.363840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr" event={"ID":"bda2d0d5-3962-4ae8-833d-e025a5fc532b","Type":"ContainerStarted","Data":"33b4d1e9ba7dcda7067af3801f280761efff7104ebb621bdd620f34cf7c5ba22"}
Sep 09 20:16:21.369562 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:21.369475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr" event={"ID":"bda2d0d5-3962-4ae8-833d-e025a5fc532b","Type":"ContainerStarted","Data":"a07048f0c51476c6df474b3c914712f3c88e76076f7b157f2c897c465aca3ef9"}
Sep 09 20:16:21.369997 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:21.369570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr" event={"ID":"bda2d0d5-3962-4ae8-833d-e025a5fc532b","Type":"ContainerStarted","Data":"c244199c2fc41344352314395a1c1562e91af9746a4129dd73dbc15db04bf2ef"}
Sep 09 20:16:21.371687 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:21.371654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdkmv" event={"ID":"51288a2a-3ab9-4f14-9930-a0105a768158","Type":"ContainerStarted","Data":"2732075dcd2c092760675e51f1478a862f0d29696c8c6d82e591efa1ea55bbab"}
Sep 09 20:16:21.385113 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:21.385044    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-hx8mr" podStartSLOduration=1.838722231 podStartE2EDuration="3.385030387s" totalImagesPullingTime="1.546308156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:21.384557237 +0000 UTC m=+297.358327299" watchObservedRunningTime="2026-09-09 20:16:21.385030387 +0000 UTC m=+297.358800452"
Sep 09 20:16:21.404380 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:21.404339    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pdkmv" podStartSLOduration=33.575527014 podStartE2EDuration="36.404325593s" totalImagesPullingTime="2.828798579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:15:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:21.403779843 +0000 UTC m=+297.377549973" watchObservedRunningTime="2026-09-09 20:16:21.404325593 +0000 UTC m=+297.378095655"
Sep 09 20:16:23.529198 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.529155    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"]
Sep 09 20:16:23.531580 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.531560    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.534056 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.534027    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:16:23.534176 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.534110    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:16:23.534176 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.534139    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nktwr\""
Sep 09 20:16:23.542450 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.542429    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"]
Sep 09 20:16:23.568081 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.568052    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2c49z"]
Sep 09 20:16:23.570160 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.570142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.572789 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.572767    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:16:23.572886 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.572832    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:16:23.573063 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.573047    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:16:23.573143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.573123    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5js7b\""
Sep 09 20:16:23.665085 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-sys\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665085 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-metrics-client-ca\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-accelerators-collector-config\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-wtmp\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.665282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c02d415d-ee36-4b41-abfc-9d7145ac0895-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.665282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-tls\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665485 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.665485 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-root\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665485 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-textfile\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.665485 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgjw\" (UniqueName: \"kubernetes.io/projected/c02d415d-ee36-4b41-abfc-9d7145ac0895-kube-api-access-rlgjw\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.665485 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.665413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcmf\" (UniqueName: \"kubernetes.io/projected/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-kube-api-access-wdcmf\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766181 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-sys\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766346 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-metrics-client-ca\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766346 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-accelerators-collector-config\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766346 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766346 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-wtmp\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766346 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.766594 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c02d415d-ee36-4b41-abfc-9d7145ac0895-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.766594 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-tls\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766594 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.766594 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-root\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766594 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-textfile\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766594 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766531    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgjw\" (UniqueName: \"kubernetes.io/projected/c02d415d-ee36-4b41-abfc-9d7145ac0895-kube-api-access-rlgjw\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.766594 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766569    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcmf\" (UniqueName: \"kubernetes.io/projected/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-kube-api-access-wdcmf\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.766901 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:23.766763    2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 20:16:23.766901 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:23.766822    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-tls podName:c02d415d-ee36-4b41-abfc-9d7145ac0895 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:24.266804372 +0000 UTC m=+300.240574418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-n2dfq" (UID: "c02d415d-ee36-4b41-abfc-9d7145ac0895") : secret "openshift-state-metrics-tls" not found
Sep 09 20:16:23.767030 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.766967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-accelerators-collector-config\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.767100 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.767069    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-root\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.767100 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:23.767089    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:16:23.767181 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.767128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-sys\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.767181 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.767089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-textfile\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.767181 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:16:23.767166    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-tls podName:a8859db1-e24a-4ce2-9962-4f7c2ae74ca0 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:24.267148829 +0000 UTC m=+300.240918870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-tls") pod "node-exporter-2c49z" (UID: "a8859db1-e24a-4ce2-9962-4f7c2ae74ca0") : secret "node-exporter-tls" not found
Sep 09 20:16:23.767303 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.767217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-wtmp\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.767709 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.767688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-metrics-client-ca\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.767807 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.767784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c02d415d-ee36-4b41-abfc-9d7145ac0895-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.769754 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.769718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.770034 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.769995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:23.776852 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.776831    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgjw\" (UniqueName: \"kubernetes.io/projected/c02d415d-ee36-4b41-abfc-9d7145ac0895-kube-api-access-rlgjw\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:23.777468 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:23.777449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcmf\" (UniqueName: \"kubernetes.io/projected/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-kube-api-access-wdcmf\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:24.269626 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.269583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-tls\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:24.269791 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.269646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:24.271974 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.271948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8859db1-e24a-4ce2-9962-4f7c2ae74ca0-node-exporter-tls\") pod \"node-exporter-2c49z\" (UID: \"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0\") " pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:24.272109 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.272078    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c02d415d-ee36-4b41-abfc-9d7145ac0895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-n2dfq\" (UID: \"c02d415d-ee36-4b41-abfc-9d7145ac0895\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:24.377692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.377654    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:16:24.380954 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.380928    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:16:24.393430 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.393408    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:16:24.443282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.443142    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nktwr\""
Sep 09 20:16:24.450858 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.450833    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"
Sep 09 20:16:24.481478 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.481445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5js7b\""
Sep 09 20:16:24.490873 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.490835    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2c49z"
Sep 09 20:16:24.503018 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:24.502962    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8859db1_e24a_4ce2_9962_4f7c2ae74ca0.slice/crio-ea76d6a9b8aba8e17277ae36f10c2fbec653cf4fee277e5170d080e46771cd6c WatchSource:0}: Error finding container ea76d6a9b8aba8e17277ae36f10c2fbec653cf4fee277e5170d080e46771cd6c: Status 404 returned error can't find the container with id ea76d6a9b8aba8e17277ae36f10c2fbec653cf4fee277e5170d080e46771cd6c
Sep 09 20:16:24.733591 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:24.733506    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq"]
Sep 09 20:16:24.736490 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:24.736466    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02d415d_ee36_4b41_abfc_9d7145ac0895.slice/crio-dbbda886bb9fd5f798658d90931f45daeae92d9d7a7115aed6fa7d770ef57e02 WatchSource:0}: Error finding container dbbda886bb9fd5f798658d90931f45daeae92d9d7a7115aed6fa7d770ef57e02: Status 404 returned error can't find the container with id dbbda886bb9fd5f798658d90931f45daeae92d9d7a7115aed6fa7d770ef57e02
Sep 09 20:16:25.390990 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:25.390939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2c49z" event={"ID":"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0","Type":"ContainerStarted","Data":"ea76d6a9b8aba8e17277ae36f10c2fbec653cf4fee277e5170d080e46771cd6c"}
Sep 09 20:16:25.394227 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:25.394192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq" event={"ID":"c02d415d-ee36-4b41-abfc-9d7145ac0895","Type":"ContainerStarted","Data":"506c8d5928cc78fe1748870d8a959b962bbaf2a9ad8f188f63d66ed729974709"}
Sep 09 20:16:25.394383 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:25.394259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq" event={"ID":"c02d415d-ee36-4b41-abfc-9d7145ac0895","Type":"ContainerStarted","Data":"160590926c2fad45e9b602f33f89a37ca0baa02244eb2109c1ff8aa96011525a"}
Sep 09 20:16:25.394383 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:25.394285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq" event={"ID":"c02d415d-ee36-4b41-abfc-9d7145ac0895","Type":"ContainerStarted","Data":"dbbda886bb9fd5f798658d90931f45daeae92d9d7a7115aed6fa7d770ef57e02"}
Sep 09 20:16:26.400151 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:26.400115    2562 generic.go:356] "Generic (PLEG): container finished" podID="a8859db1-e24a-4ce2-9962-4f7c2ae74ca0" containerID="adbc08545b6426ff8424b626c331edc633e77bbcdad0c59a6d09f68cdcf904db" exitCode=0
Sep 09 20:16:26.400637 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:26.400194    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2c49z" event={"ID":"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0","Type":"ContainerDied","Data":"adbc08545b6426ff8424b626c331edc633e77bbcdad0c59a6d09f68cdcf904db"}
Sep 09 20:16:26.402653 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:26.402625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq" event={"ID":"c02d415d-ee36-4b41-abfc-9d7145ac0895","Type":"ContainerStarted","Data":"f6697f98a943f5e5597c9c5cbdb0923db37bd87206381a01182e8c15a0f03052"}
Sep 09 20:16:26.440431 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:26.440386    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n2dfq" podStartSLOduration=2.379135976 podStartE2EDuration="3.440373191s" totalImagesPullingTime="1.061237215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:26.439920864 +0000 UTC m=+302.413690928" watchObservedRunningTime="2026-09-09 20:16:26.440373191 +0000 UTC m=+302.414143253"
Sep 09 20:16:27.408589 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:27.408552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2c49z" event={"ID":"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0","Type":"ContainerStarted","Data":"fc6c020d4bd98f239e492d828eaa795d6d8f086564446811e51f5a5926885951"}
Sep 09 20:16:27.409119 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:27.408642    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2c49z" event={"ID":"a8859db1-e24a-4ce2-9962-4f7c2ae74ca0","Type":"ContainerStarted","Data":"1f1553db5a9b111cec3d8ad631db4232697c3645725ab55ecd12feeb984a3264"}
Sep 09 20:16:27.428660 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:27.428602    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2c49z" podStartSLOduration=2.939521189 podStartE2EDuration="4.428587395s" totalImagesPullingTime="1.426173397s" totalInitContainerRuntime="62.892809ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:27.427265798 +0000 UTC m=+303.401035861" watchObservedRunningTime="2026-09-09 20:16:27.428587395 +0000 UTC m=+303.402357458"
Sep 09 20:16:29.309970 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:29.309942    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d957f96b-g54nh"
Sep 09 20:16:31.460586 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:31.460558    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:16:36.035164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.035126    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ddd7c7855-7zrml"]
Sep 09 20:16:36.037886 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.037868    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.040610 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.040588    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:16:36.041776 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.041749    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:16:36.041957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.041786    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:16:36.041957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.041798    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:16:36.041957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.041828    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tmbk2\""
Sep 09 20:16:36.041957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.041858    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:16:36.041957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.041749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:16:36.041957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.041780    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:16:36.046459 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.046434    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:16:36.047736 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.047718    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddd7c7855-7zrml"]
Sep 09 20:16:36.174195 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.174165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-config\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.174195 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.174194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-oauth-serving-cert\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.174400 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.174217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npc72\" (UniqueName: \"kubernetes.io/projected/dfa5d17e-1e06-4223-b95a-e77962fd6b61-kube-api-access-npc72\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.174400 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.174244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-service-ca\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.174400 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.174312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-serving-cert\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.174400 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.174376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-trusted-ca-bundle\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.174534 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.174413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-oauth-config\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.275603 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.275573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-service-ca\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.275603 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.275608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-serving-cert\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.275808 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.275634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-trusted-ca-bundle\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.275808 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.275661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-oauth-config\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.275808 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.275690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-config\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.275964 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.275800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-oauth-serving-cert\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.275964 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.275856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npc72\" (UniqueName: \"kubernetes.io/projected/dfa5d17e-1e06-4223-b95a-e77962fd6b61-kube-api-access-npc72\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.276448 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.276423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-config\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.276570 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.276454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-service-ca\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.276632 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.276573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-oauth-serving-cert\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.276697 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.276647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-trusted-ca-bundle\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.278228 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.278203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-oauth-config\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.278301 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.278265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-serving-cert\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.283635 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.283609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npc72\" (UniqueName: \"kubernetes.io/projected/dfa5d17e-1e06-4223-b95a-e77962fd6b61-kube-api-access-npc72\") pod \"console-ddd7c7855-7zrml\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") " pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.347884 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.347830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:36.616114 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.616034    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddd7c7855-7zrml"]
Sep 09 20:16:36.619125 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:16:36.619099    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa5d17e_1e06_4223_b95a_e77962fd6b61.slice/crio-3d8787fb7a57d3555113a2ad638b4cedacdf53836b53f82d118d6bb5ddb8b01e WatchSource:0}: Error finding container 3d8787fb7a57d3555113a2ad638b4cedacdf53836b53f82d118d6bb5ddb8b01e: Status 404 returned error can't find the container with id 3d8787fb7a57d3555113a2ad638b4cedacdf53836b53f82d118d6bb5ddb8b01e
Sep 09 20:16:36.621211 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:36.621194    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:16:37.446532 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:37.446496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddd7c7855-7zrml" event={"ID":"dfa5d17e-1e06-4223-b95a-e77962fd6b61","Type":"ContainerStarted","Data":"3d8787fb7a57d3555113a2ad638b4cedacdf53836b53f82d118d6bb5ddb8b01e"}
Sep 09 20:16:39.455653 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:39.455623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddd7c7855-7zrml" event={"ID":"dfa5d17e-1e06-4223-b95a-e77962fd6b61","Type":"ContainerStarted","Data":"a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96"}
Sep 09 20:16:39.475085 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:39.475033    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ddd7c7855-7zrml" podStartSLOduration=0.751470153 podStartE2EDuration="3.475018184s" totalImagesPullingTime="2.723548031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:39.474248042 +0000 UTC m=+315.448018106" watchObservedRunningTime="2026-09-09 20:16:39.475018184 +0000 UTC m=+315.448788252"
Sep 09 20:16:46.348420 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:46.348387    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:46.348794 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:46.348461    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:46.352986 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:46.352963    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:46.484538 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:46.484511    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:16:54.512862 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:54.512828    2562 generic.go:356] "Generic (PLEG): container finished" podID="2b1c687f-3564-4380-b568-7d619a225fe3" containerID="c6558be5c04dd49058194d5fc7efc9a5254c200ab666ab382e5a8f354519d428" exitCode=0
Sep 09 20:16:54.513260 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:54.512902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z" event={"ID":"2b1c687f-3564-4380-b568-7d619a225fe3","Type":"ContainerDied","Data":"c6558be5c04dd49058194d5fc7efc9a5254c200ab666ab382e5a8f354519d428"}
Sep 09 20:16:54.513303 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:54.513265    2562 scope.go:117] "RemoveContainer" containerID="c6558be5c04dd49058194d5fc7efc9a5254c200ab666ab382e5a8f354519d428"
Sep 09 20:16:55.518147 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:16:55.518115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-88f8z" event={"ID":"2b1c687f-3564-4380-b568-7d619a225fe3","Type":"ContainerStarted","Data":"dd9db9d220aba73daba4756777ce391c9b01693ed3e51b5d791cf5c87db18203"}
Sep 09 20:17:15.592333 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:15.592302    2562 generic.go:356] "Generic (PLEG): container finished" podID="41ae5d00-e0ab-425b-bb62-7452b59c2101" containerID="9ebb99e7bfeeb90d01165104d28c84da49ba1bf1cd81bb5ad3736886f0eb3c98" exitCode=0
Sep 09 20:17:15.592725 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:15.592370    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg" event={"ID":"41ae5d00-e0ab-425b-bb62-7452b59c2101","Type":"ContainerDied","Data":"9ebb99e7bfeeb90d01165104d28c84da49ba1bf1cd81bb5ad3736886f0eb3c98"}
Sep 09 20:17:15.592725 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:15.592672    2562 scope.go:117] "RemoveContainer" containerID="9ebb99e7bfeeb90d01165104d28c84da49ba1bf1cd81bb5ad3736886f0eb3c98"
Sep 09 20:17:16.597385 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:16.597354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xchkg" event={"ID":"41ae5d00-e0ab-425b-bb62-7452b59c2101","Type":"ContainerStarted","Data":"907e24f43d91ae5acf32e0128e28312aaa691103f8752029d3a632156a0f0cd2"}
Sep 09 20:17:38.360907 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:38.360867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:17:38.363141 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:38.363121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b355c14c-96e4-4d5d-ada5-6142cd99e951-metrics-certs\") pod \"network-metrics-daemon-zq8t9\" (UID: \"b355c14c-96e4-4d5d-ada5-6142cd99e951\") " pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:17:38.663573 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:38.663538    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hb455\""
Sep 09 20:17:38.671797 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:38.671777    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq8t9"
Sep 09 20:17:38.943474 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:38.943391    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zq8t9"]
Sep 09 20:17:38.946195 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:17:38.946161    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb355c14c_96e4_4d5d_ada5_6142cd99e951.slice/crio-7041eceefa0157a65b1235cd6cf759cc1898a09be6bac6324b789c60cb2f6c0b WatchSource:0}: Error finding container 7041eceefa0157a65b1235cd6cf759cc1898a09be6bac6324b789c60cb2f6c0b: Status 404 returned error can't find the container with id 7041eceefa0157a65b1235cd6cf759cc1898a09be6bac6324b789c60cb2f6c0b
Sep 09 20:17:39.680493 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:39.680453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq8t9" event={"ID":"b355c14c-96e4-4d5d-ada5-6142cd99e951","Type":"ContainerStarted","Data":"7041eceefa0157a65b1235cd6cf759cc1898a09be6bac6324b789c60cb2f6c0b"}
Sep 09 20:17:40.685993 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:40.685955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq8t9" event={"ID":"b355c14c-96e4-4d5d-ada5-6142cd99e951","Type":"ContainerStarted","Data":"9b798f57ec15b8a820bd7f2f3bc55ad2b74d45f86f7cc7e37bfd78ba863e6d3b"}
Sep 09 20:17:40.686412 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:40.686023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq8t9" event={"ID":"b355c14c-96e4-4d5d-ada5-6142cd99e951","Type":"ContainerStarted","Data":"e66b8aa2ff9052160655763ce35215f2c519b0f9137b096cf489181b26c6bcc5"}
Sep 09 20:17:40.702261 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:40.702219    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zq8t9" podStartSLOduration=375.879811613 podStartE2EDuration="6m16.702204583s" totalImagesPullingTime="822.39297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:40.70043887 +0000 UTC m=+376.674208955" watchObservedRunningTime="2026-09-09 20:17:40.702204583 +0000 UTC m=+376.675974646"
Sep 09 20:17:51.100217 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:17:51.100183    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ddd7c7855-7zrml"]
Sep 09 20:18:16.120280 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.120228    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ddd7c7855-7zrml" podUID="dfa5d17e-1e06-4223-b95a-e77962fd6b61" containerName="console" containerID="cri-o://a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96" gracePeriod=15
Sep 09 20:18:16.366757 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.366731    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:18:16.449042 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.448990    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-oauth-config\") pod \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") "
Sep 09 20:18:16.449239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449058    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-trusted-ca-bundle\") pod \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") "
Sep 09 20:18:16.449239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449089    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-oauth-serving-cert\") pod \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") "
Sep 09 20:18:16.449239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449125    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npc72\" (UniqueName: \"kubernetes.io/projected/dfa5d17e-1e06-4223-b95a-e77962fd6b61-kube-api-access-npc72\") pod \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") "
Sep 09 20:18:16.449239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449148    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-serving-cert\") pod \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") "
Sep 09 20:18:16.449239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449206    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-service-ca\") pod \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") "
Sep 09 20:18:16.449239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-config\") pod \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\" (UID: \"dfa5d17e-1e06-4223-b95a-e77962fd6b61\") "
Sep 09 20:18:16.449625 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449578    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dfa5d17e-1e06-4223-b95a-e77962fd6b61" (UID: "dfa5d17e-1e06-4223-b95a-e77962fd6b61"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:16.449625 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449604    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-service-ca" (OuterVolumeSpecName: "service-ca") pod "dfa5d17e-1e06-4223-b95a-e77962fd6b61" (UID: "dfa5d17e-1e06-4223-b95a-e77962fd6b61"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:16.449774 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dfa5d17e-1e06-4223-b95a-e77962fd6b61" (UID: "dfa5d17e-1e06-4223-b95a-e77962fd6b61"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:16.449774 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.449715    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-config" (OuterVolumeSpecName: "console-config") pod "dfa5d17e-1e06-4223-b95a-e77962fd6b61" (UID: "dfa5d17e-1e06-4223-b95a-e77962fd6b61"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:16.451439 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.451413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dfa5d17e-1e06-4223-b95a-e77962fd6b61" (UID: "dfa5d17e-1e06-4223-b95a-e77962fd6b61"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:16.451522 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.451471    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa5d17e-1e06-4223-b95a-e77962fd6b61-kube-api-access-npc72" (OuterVolumeSpecName: "kube-api-access-npc72") pod "dfa5d17e-1e06-4223-b95a-e77962fd6b61" (UID: "dfa5d17e-1e06-4223-b95a-e77962fd6b61"). InnerVolumeSpecName "kube-api-access-npc72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:16.451522 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.451481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dfa5d17e-1e06-4223-b95a-e77962fd6b61" (UID: "dfa5d17e-1e06-4223-b95a-e77962fd6b61"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:16.550296 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.550265    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.550296 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.550290    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.550296 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.550299    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-oauth-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.550495 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.550308    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-trusted-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.550495 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.550316    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa5d17e-1e06-4223-b95a-e77962fd6b61-oauth-serving-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.550495 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.550324    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npc72\" (UniqueName: \"kubernetes.io/projected/dfa5d17e-1e06-4223-b95a-e77962fd6b61-kube-api-access-npc72\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.550495 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.550334    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa5d17e-1e06-4223-b95a-e77962fd6b61-console-serving-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.824388 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.824304    2562 generic.go:356] "Generic (PLEG): container finished" podID="dfa5d17e-1e06-4223-b95a-e77962fd6b61" containerID="a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96" exitCode=0
Sep 09 20:18:16.824388 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.824349    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddd7c7855-7zrml"
Sep 09 20:18:16.824605 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.824396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddd7c7855-7zrml" event={"ID":"dfa5d17e-1e06-4223-b95a-e77962fd6b61","Type":"ContainerDied","Data":"a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96"}
Sep 09 20:18:16.824605 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.824450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddd7c7855-7zrml" event={"ID":"dfa5d17e-1e06-4223-b95a-e77962fd6b61","Type":"ContainerDied","Data":"3d8787fb7a57d3555113a2ad638b4cedacdf53836b53f82d118d6bb5ddb8b01e"}
Sep 09 20:18:16.824605 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.824483    2562 scope.go:117] "RemoveContainer" containerID="a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96"
Sep 09 20:18:16.832821 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.832804    2562 scope.go:117] "RemoveContainer" containerID="a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96"
Sep 09 20:18:16.833109 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:18:16.833089    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96\": container with ID starting with a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96 not found: ID does not exist" containerID="a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96"
Sep 09 20:18:16.833164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.833116    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96"} err="failed to get container status \"a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96\": rpc error: code = NotFound desc = could not find container \"a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96\": container with ID starting with a9ec3c0f17b1ad91bd16ed519decad9043d7ac210834f59a2817bf90b2959d96 not found: ID does not exist"
Sep 09 20:18:16.839788 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.839766    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ddd7c7855-7zrml"]
Sep 09 20:18:16.843604 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:16.843585    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ddd7c7855-7zrml"]
Sep 09 20:18:18.465604 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:18:18.465573    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa5d17e-1e06-4223-b95a-e77962fd6b61" path="/var/lib/kubelet/pods/dfa5d17e-1e06-4223-b95a-e77962fd6b61/volumes"
Sep 09 20:20:11.053907 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.053872    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"]
Sep 09 20:20:11.054418 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.054179    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa5d17e-1e06-4223-b95a-e77962fd6b61" containerName="console"
Sep 09 20:20:11.054418 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.054194    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa5d17e-1e06-4223-b95a-e77962fd6b61" containerName="console"
Sep 09 20:20:11.054418 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.054255    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa5d17e-1e06-4223-b95a-e77962fd6b61" containerName="console"
Sep 09 20:20:11.057127 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.057108    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.059852 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.059825    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:20:11.059972 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.059834    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:20:11.059972 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.059864    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-gjqxn\""
Sep 09 20:20:11.060162 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.060146    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:20:11.074383 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.074354    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"]
Sep 09 20:20:11.169633 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.169591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljhm\" (UniqueName: \"kubernetes.io/projected/be9ca3ed-ac2e-431d-b3e7-f114e0ca9620-kube-api-access-lljhm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dw55s\" (UID: \"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.169804 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.169656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/be9ca3ed-ac2e-431d-b3e7-f114e0ca9620-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dw55s\" (UID: \"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.270228 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.270172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/be9ca3ed-ac2e-431d-b3e7-f114e0ca9620-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dw55s\" (UID: \"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.270406 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.270272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljhm\" (UniqueName: \"kubernetes.io/projected/be9ca3ed-ac2e-431d-b3e7-f114e0ca9620-kube-api-access-lljhm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dw55s\" (UID: \"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.272602 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.272573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/be9ca3ed-ac2e-431d-b3e7-f114e0ca9620-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dw55s\" (UID: \"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.282748 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.280250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljhm\" (UniqueName: \"kubernetes.io/projected/be9ca3ed-ac2e-431d-b3e7-f114e0ca9620-kube-api-access-lljhm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dw55s\" (UID: \"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.367464 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.367360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:11.663545 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:11.663507    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"]
Sep 09 20:20:11.666629 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:20:11.666603    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9ca3ed_ac2e_431d_b3e7_f114e0ca9620.slice/crio-c5afb136ded4d6cb51b977833f2a94b1bd9b0e4fb378bf1233434897a4d9ede4 WatchSource:0}: Error finding container c5afb136ded4d6cb51b977833f2a94b1bd9b0e4fb378bf1233434897a4d9ede4: Status 404 returned error can't find the container with id c5afb136ded4d6cb51b977833f2a94b1bd9b0e4fb378bf1233434897a4d9ede4
Sep 09 20:20:12.241422 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:12.241388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s" event={"ID":"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620","Type":"ContainerStarted","Data":"c5afb136ded4d6cb51b977833f2a94b1bd9b0e4fb378bf1233434897a4d9ede4"}
Sep 09 20:20:15.259091 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.258984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s" event={"ID":"be9ca3ed-ac2e-431d-b3e7-f114e0ca9620","Type":"ContainerStarted","Data":"1d207dbac5ace5299e1c88fdb6b3489002d011da8aef7cf76c71b8008ee013c5"}
Sep 09 20:20:15.259475 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.259134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:20:15.282755 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.282694    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s" podStartSLOduration=1.032063613 podStartE2EDuration="4.282677365s" totalImagesPullingTime="3.250613752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:15.281644538 +0000 UTC m=+531.255414601" watchObservedRunningTime="2026-09-09 20:20:15.282677365 +0000 UTC m=+531.256447428"
Sep 09 20:20:15.814678 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.814640    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"]
Sep 09 20:20:15.823562 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.823527    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:15.827094 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.827064    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:20:15.827216 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.827071    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7l8zv\""
Sep 09 20:20:15.827292 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.827276    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:20:15.837352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.837328    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"]
Sep 09 20:20:15.912452 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.912415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:15.912452 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.912452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq92h\" (UniqueName: \"kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-kube-api-access-nq92h\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:15.912693 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:15.912541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/730cbfe6-771d-45aa-b5e4-016754b9101f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.013728 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.013696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/730cbfe6-771d-45aa-b5e4-016754b9101f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.013906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.013755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.013906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.013780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq92h\" (UniqueName: \"kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-kube-api-access-nq92h\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.013906 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:20:16.013868    2562 secret.go:281] references non-existent secret key: tls.crt
Sep 09 20:20:16.013906 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:20:16.013888    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 09 20:20:16.013906 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:20:16.013906    2562 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 09 20:20:16.014205 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:20:16.013928    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 09 20:20:16.014205 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:20:16.013991    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-certificates podName:730cbfe6-771d-45aa-b5e4-016754b9101f nodeName:}" failed. No retries permitted until 2026-09-09 20:20:16.513971476 +0000 UTC m=+532.487741531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-certificates") pod "keda-metrics-apiserver-77857b7b98-v9z7p" (UID: "730cbfe6-771d-45aa-b5e4-016754b9101f") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 09 20:20:16.014205 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.014181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/730cbfe6-771d-45aa-b5e4-016754b9101f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.022795 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.022757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq92h\" (UniqueName: \"kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-kube-api-access-nq92h\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.519669 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.519630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.522170 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.522136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/730cbfe6-771d-45aa-b5e4-016754b9101f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-v9z7p\" (UID: \"730cbfe6-771d-45aa-b5e4-016754b9101f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:16.734333 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:16.734293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:17.009582 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:17.009498    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"]
Sep 09 20:20:17.013246 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:20:17.013212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730cbfe6_771d_45aa_b5e4_016754b9101f.slice/crio-56fb8fba63fe08ffecd19429977e2d13a8de9926943b540252c0573f44f4e91d WatchSource:0}: Error finding container 56fb8fba63fe08ffecd19429977e2d13a8de9926943b540252c0573f44f4e91d: Status 404 returned error can't find the container with id 56fb8fba63fe08ffecd19429977e2d13a8de9926943b540252c0573f44f4e91d
Sep 09 20:20:17.268868 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:17.268834    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p" event={"ID":"730cbfe6-771d-45aa-b5e4-016754b9101f","Type":"ContainerStarted","Data":"56fb8fba63fe08ffecd19429977e2d13a8de9926943b540252c0573f44f4e91d"}
Sep 09 20:20:20.285399 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:20.285365    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p" event={"ID":"730cbfe6-771d-45aa-b5e4-016754b9101f","Type":"ContainerStarted","Data":"5cbf03a1701ae133033cc27d8120cadd2ba84bdd2a06e3aa3f44ee5b159a792a"}
Sep 09 20:20:20.285775 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:20.285479    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:20.302679 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:20.302618    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p" podStartSLOduration=2.7074942269999998 podStartE2EDuration="5.302594908s" totalImagesPullingTime="2.595100681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:20.302129882 +0000 UTC m=+536.275899946" watchObservedRunningTime="2026-09-09 20:20:20.302594908 +0000 UTC m=+536.276364971"
Sep 09 20:20:31.294670 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:31.294637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-v9z7p"
Sep 09 20:20:36.265686 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:20:36.265650    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dw55s"
Sep 09 20:21:22.735957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.735915    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c8b7c86-dpgnm"]
Sep 09 20:21:22.739332 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.739306    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.743342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743288    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:21:22.743342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743305    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:21:22.743342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743288    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:21:22.743342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743321    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:21:22.743721 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tmbk2\""
Sep 09 20:21:22.743721 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743694    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:21:22.743822 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743810    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:21:22.743909 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.743838    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:21:22.748982 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.748959    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:21:22.753363 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.753341    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c8b7c86-dpgnm"]
Sep 09 20:21:22.858211 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.858177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-trusted-ca-bundle\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.858370 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.858219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-service-ca\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.858370 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.858252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-oauth-serving-cert\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.858370 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.858294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-serving-cert\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.858467 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.858370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-config\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.858467 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.858434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-oauth-config\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.858467 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.858465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhh2\" (UniqueName: \"kubernetes.io/projected/5ff147f7-a7f1-4ea2-a9ea-237f78482196-kube-api-access-pnhh2\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.959543 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.959500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-trusted-ca-bundle\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.959543 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.959550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-service-ca\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.959751 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.959584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-oauth-serving-cert\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.959751 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.959608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-serving-cert\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.959751 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.959657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-config\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.959751 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.959710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-oauth-config\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.959751 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.959745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhh2\" (UniqueName: \"kubernetes.io/projected/5ff147f7-a7f1-4ea2-a9ea-237f78482196-kube-api-access-pnhh2\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.960458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.960426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-oauth-serving-cert\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.960458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.960436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-service-ca\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.960646 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.960540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-config\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.960646 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.960623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff147f7-a7f1-4ea2-a9ea-237f78482196-trusted-ca-bundle\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.962289 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.962264    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-oauth-config\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.962383 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.962272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ff147f7-a7f1-4ea2-a9ea-237f78482196-console-serving-cert\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:22.969582 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:22.969558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhh2\" (UniqueName: \"kubernetes.io/projected/5ff147f7-a7f1-4ea2-a9ea-237f78482196-kube-api-access-pnhh2\") pod \"console-75c8b7c86-dpgnm\" (UID: \"5ff147f7-a7f1-4ea2-a9ea-237f78482196\") " pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:23.049670 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:23.049571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:23.327570 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:23.327545    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c8b7c86-dpgnm"]
Sep 09 20:21:23.330174 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:21:23.330144    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff147f7_a7f1_4ea2_a9ea_237f78482196.slice/crio-50e5f45fd1eebe7f5b978c1fbbb7d697b5d75727bfc2b8ee7db2461c06097312 WatchSource:0}: Error finding container 50e5f45fd1eebe7f5b978c1fbbb7d697b5d75727bfc2b8ee7db2461c06097312: Status 404 returned error can't find the container with id 50e5f45fd1eebe7f5b978c1fbbb7d697b5d75727bfc2b8ee7db2461c06097312
Sep 09 20:21:23.546840 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:23.546794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c8b7c86-dpgnm" event={"ID":"5ff147f7-a7f1-4ea2-a9ea-237f78482196","Type":"ContainerStarted","Data":"6847135fa6c3247fb9f2dd7faa3d2bfa2c552e9508d9843a5ea33c4f4759f7f2"}
Sep 09 20:21:23.546840 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:23.546848    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c8b7c86-dpgnm" event={"ID":"5ff147f7-a7f1-4ea2-a9ea-237f78482196","Type":"ContainerStarted","Data":"50e5f45fd1eebe7f5b978c1fbbb7d697b5d75727bfc2b8ee7db2461c06097312"}
Sep 09 20:21:23.565558 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:23.565514    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75c8b7c86-dpgnm" podStartSLOduration=1.5655005800000001 podStartE2EDuration="1.56550058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:23.564354721 +0000 UTC m=+599.538124806" watchObservedRunningTime="2026-09-09 20:21:23.56550058 +0000 UTC m=+599.539270642"
Sep 09 20:21:24.407270 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:24.407240    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:21:24.407790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:24.407763    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:21:33.050211 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:33.050162    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:33.052714 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:33.050314    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:33.055184 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:33.055164    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:21:33.590408 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:21:33.590377    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75c8b7c86-dpgnm"
Sep 09 20:22:53.690167 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.690137    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-98pmb"]
Sep 09 20:22:53.693186 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.693170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:53.696915 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.696890    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:22:53.696915 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.696898    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z42zl\""
Sep 09 20:22:53.697125 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.696962    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:22:53.697328 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.697308    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:22:53.706864 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.706841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-98pmb"]
Sep 09 20:22:53.751279 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.751253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5grd\" (UniqueName: \"kubernetes.io/projected/93854588-f364-425a-9a4d-c7b41809582d-kube-api-access-k5grd\") pod \"model-serving-api-7b954ffc95-98pmb\" (UID: \"93854588-f364-425a-9a4d-c7b41809582d\") " pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:53.751376 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.751281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93854588-f364-425a-9a4d-c7b41809582d-tls-certs\") pod \"model-serving-api-7b954ffc95-98pmb\" (UID: \"93854588-f364-425a-9a4d-c7b41809582d\") " pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:53.852264 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.852162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5grd\" (UniqueName: \"kubernetes.io/projected/93854588-f364-425a-9a4d-c7b41809582d-kube-api-access-k5grd\") pod \"model-serving-api-7b954ffc95-98pmb\" (UID: \"93854588-f364-425a-9a4d-c7b41809582d\") " pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:53.852264 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.852200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93854588-f364-425a-9a4d-c7b41809582d-tls-certs\") pod \"model-serving-api-7b954ffc95-98pmb\" (UID: \"93854588-f364-425a-9a4d-c7b41809582d\") " pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:53.854527 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.854507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93854588-f364-425a-9a4d-c7b41809582d-tls-certs\") pod \"model-serving-api-7b954ffc95-98pmb\" (UID: \"93854588-f364-425a-9a4d-c7b41809582d\") " pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:53.860186 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:53.860166    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5grd\" (UniqueName: \"kubernetes.io/projected/93854588-f364-425a-9a4d-c7b41809582d-kube-api-access-k5grd\") pod \"model-serving-api-7b954ffc95-98pmb\" (UID: \"93854588-f364-425a-9a4d-c7b41809582d\") " pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:54.003208 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:54.003121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:54.274493 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:54.274432    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-98pmb"]
Sep 09 20:22:54.276767 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:22:54.276737    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93854588_f364_425a_9a4d_c7b41809582d.slice/crio-2092eae481fccb0a9fdfed612d9c7cdd2246c88ed7e06fa48c91952a50f30c1e WatchSource:0}: Error finding container 2092eae481fccb0a9fdfed612d9c7cdd2246c88ed7e06fa48c91952a50f30c1e: Status 404 returned error can't find the container with id 2092eae481fccb0a9fdfed612d9c7cdd2246c88ed7e06fa48c91952a50f30c1e
Sep 09 20:22:54.278552 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:54.278535    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:22:54.915762 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:54.915725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-98pmb" event={"ID":"93854588-f364-425a-9a4d-c7b41809582d","Type":"ContainerStarted","Data":"2092eae481fccb0a9fdfed612d9c7cdd2246c88ed7e06fa48c91952a50f30c1e"}
Sep 09 20:22:55.921059 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:55.920996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-98pmb" event={"ID":"93854588-f364-425a-9a4d-c7b41809582d","Type":"ContainerStarted","Data":"d8458f171da97cf13eae0e684c14a327bd1d40179a7cc2ed361f579509891223"}
Sep 09 20:22:55.921409 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:55.921178    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:22:55.938060 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:22:55.938017    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-98pmb" podStartSLOduration=1.394898299 podStartE2EDuration="2.937986476s" totalImagesPullingTime="1.543088177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:55.937722456 +0000 UTC m=+691.911492517" watchObservedRunningTime="2026-09-09 20:22:55.937986476 +0000 UTC m=+691.911756538"
Sep 09 20:23:06.930428 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:23:06.930393    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-98pmb"
Sep 09 20:24:41.050352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.050319    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"]
Sep 09 20:24:41.053579 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.053561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.056136 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.056116    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-510d5-kube-rbac-proxy-sar-config\""
Sep 09 20:24:41.056252 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.056115    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4tmvf\""
Sep 09 20:24:41.056252 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.056119    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:24:41.057016 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.056986    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-510d5-predictor-serving-cert\""
Sep 09 20:24:41.057096 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.057049    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:24:41.061806 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.061786    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"]
Sep 09 20:24:41.200373 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.200337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25c2112d-2c84-404e-a697-caf13b8d9195-proxy-tls\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.200542 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.200387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx78s\" (UniqueName: \"kubernetes.io/projected/25c2112d-2c84-404e-a697-caf13b8d9195-kube-api-access-mx78s\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.200542 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.200472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25c2112d-2c84-404e-a697-caf13b8d9195-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.301127 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.301046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25c2112d-2c84-404e-a697-caf13b8d9195-proxy-tls\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.301127 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.301107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx78s\" (UniqueName: \"kubernetes.io/projected/25c2112d-2c84-404e-a697-caf13b8d9195-kube-api-access-mx78s\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.301337 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.301157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25c2112d-2c84-404e-a697-caf13b8d9195-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.301831 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.301807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25c2112d-2c84-404e-a697-caf13b8d9195-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.303561 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.303539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25c2112d-2c84-404e-a697-caf13b8d9195-proxy-tls\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.309809 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.309786    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx78s\" (UniqueName: \"kubernetes.io/projected/25c2112d-2c84-404e-a697-caf13b8d9195-kube-api-access-mx78s\") pod \"success-200-isvc-510d5-predictor-8556d94969-wh62f\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") " pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.364068 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.364036    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:41.543141 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.543105    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"]
Sep 09 20:24:41.546870 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.546854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.549766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.549741    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 09 20:24:41.549877 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.549800    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:24:41.549877 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.549834    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:24:41.550107 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.550092    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 09 20:24:41.558419 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.558393    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"]
Sep 09 20:24:41.639669 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.639648    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"]
Sep 09 20:24:41.641911 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:24:41.641882    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c2112d_2c84_404e_a697_caf13b8d9195.slice/crio-4397398ed35471ca73010c03f336a6e347016a94a5f2c4b73ca1656ef62ad37f WatchSource:0}: Error finding container 4397398ed35471ca73010c03f336a6e347016a94a5f2c4b73ca1656ef62ad37f: Status 404 returned error can't find the container with id 4397398ed35471ca73010c03f336a6e347016a94a5f2c4b73ca1656ef62ad37f
Sep 09 20:24:41.705270 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.705232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9q5\" (UniqueName: \"kubernetes.io/projected/53132e26-85c8-4266-a005-b79dec2091c9-kube-api-access-6p9q5\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.705391 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.705280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.705391 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.705321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53132e26-85c8-4266-a005-b79dec2091c9-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.705391 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.705349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.705503 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.705443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.806730 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.806661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.806730 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.806697    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9q5\" (UniqueName: \"kubernetes.io/projected/53132e26-85c8-4266-a005-b79dec2091c9-kube-api-access-6p9q5\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.806730 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.806713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.806924 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.806785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53132e26-85c8-4266-a005-b79dec2091c9-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.806924 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:24:41.806817    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 09 20:24:41.806924 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.806827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.806924 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:24:41.806863    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls podName:53132e26-85c8-4266-a005-b79dec2091c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:24:42.306847639 +0000 UTC m=+798.280617680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-ksclf" (UID: "53132e26-85c8-4266-a005-b79dec2091c9") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 09 20:24:41.807297 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.807277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53132e26-85c8-4266-a005-b79dec2091c9-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.807525 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.807507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.807595 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.807565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:41.817577 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:41.817549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9q5\" (UniqueName: \"kubernetes.io/projected/53132e26-85c8-4266-a005-b79dec2091c9-kube-api-access-6p9q5\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:42.311162 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:42.311122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:42.315059 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:42.314995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ksclf\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:42.367977 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:42.367896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" event={"ID":"25c2112d-2c84-404e-a697-caf13b8d9195","Type":"ContainerStarted","Data":"4397398ed35471ca73010c03f336a6e347016a94a5f2c4b73ca1656ef62ad37f"}
Sep 09 20:24:42.458546 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:42.458236    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:24:42.770700 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:42.770676    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"]
Sep 09 20:24:42.776304 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:24:42.776272    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53132e26_85c8_4266_a005_b79dec2091c9.slice/crio-faeca9a3f412630196e4218ea3a8e5087cc465345c2d6229580d99435aaac591 WatchSource:0}: Error finding container faeca9a3f412630196e4218ea3a8e5087cc465345c2d6229580d99435aaac591: Status 404 returned error can't find the container with id faeca9a3f412630196e4218ea3a8e5087cc465345c2d6229580d99435aaac591
Sep 09 20:24:43.380759 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:43.380606    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerStarted","Data":"faeca9a3f412630196e4218ea3a8e5087cc465345c2d6229580d99435aaac591"}
Sep 09 20:24:56.450909 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:56.450759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerStarted","Data":"bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd"}
Sep 09 20:24:56.452913 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:56.452884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" event={"ID":"25c2112d-2c84-404e-a697-caf13b8d9195","Type":"ContainerStarted","Data":"82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d"}
Sep 09 20:24:58.466498 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:58.466460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" event={"ID":"25c2112d-2c84-404e-a697-caf13b8d9195","Type":"ContainerStarted","Data":"a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf"}
Sep 09 20:24:58.466862 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:58.466571    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:58.486164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:58.486118    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podStartSLOduration=1.416751045 podStartE2EDuration="17.486103309s" totalImagesPullingTime="16.069352264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:58.483737499 +0000 UTC m=+814.457507561" watchObservedRunningTime="2026-09-09 20:24:58.486103309 +0000 UTC m=+814.459873372"
Sep 09 20:24:59.472542 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:59.472504    2562 generic.go:356] "Generic (PLEG): container finished" podID="53132e26-85c8-4266-a005-b79dec2091c9" containerID="bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd" exitCode=0
Sep 09 20:24:59.472898 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:59.472574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerDied","Data":"bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd"}
Sep 09 20:24:59.472966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:59.472938    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:24:59.474144 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:24:59.474109    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:25:00.478788 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:00.478604    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:25:05.483990 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:05.483958    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:25:05.484514 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:05.484485    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:25:15.484514 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:15.484473    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:25:18.580574 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:18.580539    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerStarted","Data":"50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15"}
Sep 09 20:25:18.580951 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:18.580606    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerStarted","Data":"3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16"}
Sep 09 20:25:18.580951 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:18.580797    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:25:18.600361 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:18.600309    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podStartSLOduration=3.109599378 podStartE2EDuration="37.600295324s" totalImagesPullingTime="30.659094684s" totalInitContainerRuntime="3.831601262s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:18.598113316 +0000 UTC m=+834.571883390" watchObservedRunningTime="2026-09-09 20:25:18.600295324 +0000 UTC m=+834.574065387"
Sep 09 20:25:19.585579 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:19.585548    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:25:19.586947 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:19.586918    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:20.589970 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:20.589929    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:25.485406 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:25.485364    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:25:25.595075 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:25.595040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:25:25.595553 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:25.595526    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:35.484853 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:35.484808    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:25:35.595818 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:35.595783    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:45.485139 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:45.485105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:25:45.596448 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:45.596409    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:55.595666 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:25:55.595627    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:26:01.112221 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.112190    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"]
Sep 09 20:26:01.114696 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.114680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.117282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.117264    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-510d5-kube-rbac-proxy-sar-config\""
Sep 09 20:26:01.117387 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.117267    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-510d5-serving-cert\""
Sep 09 20:26:01.125111 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.125089    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"]
Sep 09 20:26:01.195171 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.195144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-openshift-service-ca-bundle\") pod \"switch-graph-510d5-85cb7dd998-8x2xs\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") " pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.195171 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.195175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-proxy-tls\") pod \"switch-graph-510d5-85cb7dd998-8x2xs\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") " pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.296427 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.296396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-openshift-service-ca-bundle\") pod \"switch-graph-510d5-85cb7dd998-8x2xs\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") " pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.296427 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.296429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-proxy-tls\") pod \"switch-graph-510d5-85cb7dd998-8x2xs\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") " pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.297013 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.296979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-openshift-service-ca-bundle\") pod \"switch-graph-510d5-85cb7dd998-8x2xs\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") " pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.298850 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.298829    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-proxy-tls\") pod \"switch-graph-510d5-85cb7dd998-8x2xs\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") " pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.426082 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.426049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:01.691096 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.690989    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"]
Sep 09 20:26:01.693770 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:26:01.693743    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6605a9e0_11e4_43e4_85ce_77d0ae9bba6d.slice/crio-2ed90eff9b38c07060549a4aade3ef08751a63987974fe6a99fe4a879e0e7afd WatchSource:0}: Error finding container 2ed90eff9b38c07060549a4aade3ef08751a63987974fe6a99fe4a879e0e7afd: Status 404 returned error can't find the container with id 2ed90eff9b38c07060549a4aade3ef08751a63987974fe6a99fe4a879e0e7afd
Sep 09 20:26:01.764060 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:01.764026    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" event={"ID":"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d","Type":"ContainerStarted","Data":"2ed90eff9b38c07060549a4aade3ef08751a63987974fe6a99fe4a879e0e7afd"}
Sep 09 20:26:03.774016 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:03.773968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" event={"ID":"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d","Type":"ContainerStarted","Data":"4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b"}
Sep 09 20:26:03.774493 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:03.774111    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:03.789276 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:03.789229    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podStartSLOduration=0.818326326 podStartE2EDuration="2.789216053s" totalImagesPullingTime="1.970889727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:03.788414443 +0000 UTC m=+879.762184504" watchObservedRunningTime="2026-09-09 20:26:03.789216053 +0000 UTC m=+879.762986115"
Sep 09 20:26:05.595801 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:05.595760    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:26:09.784010 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:09.783979    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:11.254379 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.254345    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"]
Sep 09 20:26:11.254759 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.254532    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5" containerID="cri-o://4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b" gracePeriod=30
Sep 09 20:26:11.365028 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.364983    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"]
Sep 09 20:26:11.365241 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.365218    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container" containerID="cri-o://82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d" gracePeriod=30
Sep 09 20:26:11.365322 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.365299    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kube-rbac-proxy" containerID="cri-o://a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf" gracePeriod=30
Sep 09 20:26:11.477992 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.477963    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"]
Sep 09 20:26:11.480777 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.480738    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.483275 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.483257    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e273-predictor-serving-cert\""
Sep 09 20:26:11.483275 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.483265    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e273-kube-rbac-proxy-sar-config\""
Sep 09 20:26:11.489319 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.489294    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"]
Sep 09 20:26:11.588792 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.588711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.588792 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.588751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snh7x\" (UniqueName: \"kubernetes.io/projected/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-api-access-snh7x\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.588985 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.588832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-proxy-tls\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.689242 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.689209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.689405 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.689257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snh7x\" (UniqueName: \"kubernetes.io/projected/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-api-access-snh7x\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.689405 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.689317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-proxy-tls\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.689940 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.689917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.691562 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.691541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-proxy-tls\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.697250 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.697225    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snh7x\" (UniqueName: \"kubernetes.io/projected/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-api-access-snh7x\") pod \"success-200-isvc-2e273-predictor-85c84c9849-pcfd9\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") " pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.790823 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.790791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:11.810880 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.810844    2562 generic.go:356] "Generic (PLEG): container finished" podID="25c2112d-2c84-404e-a697-caf13b8d9195" containerID="a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf" exitCode=2
Sep 09 20:26:11.811025 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:11.810919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" event={"ID":"25c2112d-2c84-404e-a697-caf13b8d9195","Type":"ContainerDied","Data":"a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf"}
Sep 09 20:26:12.058441 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:12.058407    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"]
Sep 09 20:26:12.061608 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:26:12.061584    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea26b1b_7215_49c1_8a18_5c91d850e6bd.slice/crio-b7c2e13dd8299f05412d786dd48e5fafa02f8007f97de8c8b139e0f69c8a4350 WatchSource:0}: Error finding container b7c2e13dd8299f05412d786dd48e5fafa02f8007f97de8c8b139e0f69c8a4350: Status 404 returned error can't find the container with id b7c2e13dd8299f05412d786dd48e5fafa02f8007f97de8c8b139e0f69c8a4350
Sep 09 20:26:12.818798 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:12.818764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" event={"ID":"9ea26b1b-7215-49c1-8a18-5c91d850e6bd","Type":"ContainerStarted","Data":"ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71"}
Sep 09 20:26:12.819197 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:12.818830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" event={"ID":"9ea26b1b-7215-49c1-8a18-5c91d850e6bd","Type":"ContainerStarted","Data":"98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79"}
Sep 09 20:26:12.819197 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:12.818862    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:12.819197 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:12.818880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" event={"ID":"9ea26b1b-7215-49c1-8a18-5c91d850e6bd","Type":"ContainerStarted","Data":"b7c2e13dd8299f05412d786dd48e5fafa02f8007f97de8c8b139e0f69c8a4350"}
Sep 09 20:26:12.836876 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:12.836822    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podStartSLOduration=1.836808443 podStartE2EDuration="1.836808443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:12.835215718 +0000 UTC m=+888.808985782" watchObservedRunningTime="2026-09-09 20:26:12.836808443 +0000 UTC m=+888.810578506"
Sep 09 20:26:13.823680 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:13.823646    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:13.825142 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:13.825115    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:26:14.305525 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.305494    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:26:14.410560 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.410529    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25c2112d-2c84-404e-a697-caf13b8d9195-kube-rbac-proxy-sar-config\") pod \"25c2112d-2c84-404e-a697-caf13b8d9195\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") "
Sep 09 20:26:14.410742 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.410594    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx78s\" (UniqueName: \"kubernetes.io/projected/25c2112d-2c84-404e-a697-caf13b8d9195-kube-api-access-mx78s\") pod \"25c2112d-2c84-404e-a697-caf13b8d9195\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") "
Sep 09 20:26:14.410742 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.410639    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25c2112d-2c84-404e-a697-caf13b8d9195-proxy-tls\") pod \"25c2112d-2c84-404e-a697-caf13b8d9195\" (UID: \"25c2112d-2c84-404e-a697-caf13b8d9195\") "
Sep 09 20:26:14.410956 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.410930    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c2112d-2c84-404e-a697-caf13b8d9195-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25c2112d-2c84-404e-a697-caf13b8d9195" (UID: "25c2112d-2c84-404e-a697-caf13b8d9195"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:14.412864 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.412832    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c2112d-2c84-404e-a697-caf13b8d9195-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25c2112d-2c84-404e-a697-caf13b8d9195" (UID: "25c2112d-2c84-404e-a697-caf13b8d9195"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:14.412959 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.412862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c2112d-2c84-404e-a697-caf13b8d9195-kube-api-access-mx78s" (OuterVolumeSpecName: "kube-api-access-mx78s") pod "25c2112d-2c84-404e-a697-caf13b8d9195" (UID: "25c2112d-2c84-404e-a697-caf13b8d9195"). InnerVolumeSpecName "kube-api-access-mx78s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:14.511282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.511250    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx78s\" (UniqueName: \"kubernetes.io/projected/25c2112d-2c84-404e-a697-caf13b8d9195-kube-api-access-mx78s\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.511282 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.511277    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25c2112d-2c84-404e-a697-caf13b8d9195-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.511409 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.511290    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25c2112d-2c84-404e-a697-caf13b8d9195-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.782266 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.782182    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:14.829299 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.829257    2562 generic.go:356] "Generic (PLEG): container finished" podID="25c2112d-2c84-404e-a697-caf13b8d9195" containerID="82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d" exitCode=0
Sep 09 20:26:14.829667 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.829327    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"
Sep 09 20:26:14.829667 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.829345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" event={"ID":"25c2112d-2c84-404e-a697-caf13b8d9195","Type":"ContainerDied","Data":"82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d"}
Sep 09 20:26:14.829667 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.829407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f" event={"ID":"25c2112d-2c84-404e-a697-caf13b8d9195","Type":"ContainerDied","Data":"4397398ed35471ca73010c03f336a6e347016a94a5f2c4b73ca1656ef62ad37f"}
Sep 09 20:26:14.829667 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.829434    2562 scope.go:117] "RemoveContainer" containerID="a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf"
Sep 09 20:26:14.830085 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.830049    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:26:14.838051 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.838034    2562 scope.go:117] "RemoveContainer" containerID="82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d"
Sep 09 20:26:14.845517 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.845497    2562 scope.go:117] "RemoveContainer" containerID="a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf"
Sep 09 20:26:14.845789 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:26:14.845770    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf\": container with ID starting with a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf not found: ID does not exist" containerID="a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf"
Sep 09 20:26:14.845842 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.845797    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf"} err="failed to get container status \"a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf\": rpc error: code = NotFound desc = could not find container \"a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf\": container with ID starting with a11a5f523602fc618bf6cc30e18128fa25485d92a4748103779e5374df0debbf not found: ID does not exist"
Sep 09 20:26:14.845842 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.845812    2562 scope.go:117] "RemoveContainer" containerID="82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d"
Sep 09 20:26:14.846170 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:26:14.846153    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d\": container with ID starting with 82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d not found: ID does not exist" containerID="82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d"
Sep 09 20:26:14.846231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.846174    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d"} err="failed to get container status \"82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d\": rpc error: code = NotFound desc = could not find container \"82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d\": container with ID starting with 82fd368d0560b55b1c89cef5edd13f17cd5e2dbf5827d3409c5e2301137ece1d not found: ID does not exist"
Sep 09 20:26:14.846680 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.846662    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"]
Sep 09 20:26:14.849120 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:14.849100    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-510d5-predictor-8556d94969-wh62f"]
Sep 09 20:26:15.595529 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:15.595493    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:26:16.466175 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:16.466145    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" path="/var/lib/kubelet/pods/25c2112d-2c84-404e-a697-caf13b8d9195/volumes"
Sep 09 20:26:19.782579 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:19.782541    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:19.833966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:19.833940    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:26:19.834386 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:19.834352    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:26:24.434899 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:24.434873    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:26:24.437252 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:24.437229    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:26:24.781958 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:24.781878    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:24.782111 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:24.782018    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:25.596484 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:25.596456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:26:29.782704 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:29.782669    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:29.834766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:29.834718    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:26:34.781873 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:34.781837    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:39.782133 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:39.782095    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:39.834927 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:39.834893    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:26:41.891575 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.891550    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:41.930604 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.930574    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-openshift-service-ca-bundle\") pod \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") "
Sep 09 20:26:41.930746 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.930633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-proxy-tls\") pod \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\" (UID: \"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d\") "
Sep 09 20:26:41.930961 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.930933    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" (UID: "6605a9e0-11e4-43e4-85ce-77d0ae9bba6d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:41.932601 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.932580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" (UID: "6605a9e0-11e4-43e4-85ce-77d0ae9bba6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:41.949443 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.949418    2562 generic.go:356] "Generic (PLEG): container finished" podID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerID="4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b" exitCode=0
Sep 09 20:26:41.949518 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.949456    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"
Sep 09 20:26:41.949518 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.949504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" event={"ID":"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d","Type":"ContainerDied","Data":"4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b"}
Sep 09 20:26:41.949590 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.949549    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs" event={"ID":"6605a9e0-11e4-43e4-85ce-77d0ae9bba6d","Type":"ContainerDied","Data":"2ed90eff9b38c07060549a4aade3ef08751a63987974fe6a99fe4a879e0e7afd"}
Sep 09 20:26:41.949590 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.949571    2562 scope.go:117] "RemoveContainer" containerID="4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b"
Sep 09 20:26:41.957555 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.957538    2562 scope.go:117] "RemoveContainer" containerID="4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b"
Sep 09 20:26:41.957804 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:26:41.957781    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b\": container with ID starting with 4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b not found: ID does not exist" containerID="4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b"
Sep 09 20:26:41.957887 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.957810    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b"} err="failed to get container status \"4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b\": rpc error: code = NotFound desc = could not find container \"4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b\": container with ID starting with 4b060d6eac2c69728a0beebc6aee22ac0dd949380a5e6077df399cf4c50ce67b not found: ID does not exist"
Sep 09 20:26:41.968882 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.968861    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"]
Sep 09 20:26:41.974563 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:41.974544    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-510d5-85cb7dd998-8x2xs"]
Sep 09 20:26:42.031152 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:42.031099    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:26:42.031152 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:42.031121    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:26:42.465371 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:42.465340    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" path="/var/lib/kubelet/pods/6605a9e0-11e4-43e4-85ce-77d0ae9bba6d/volumes"
Sep 09 20:26:49.834340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:49.834300    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:26:51.084896 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.084863    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"]
Sep 09 20:26:51.085272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085215    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container"
Sep 09 20:26:51.085272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085227    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container"
Sep 09 20:26:51.085272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085236    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kube-rbac-proxy"
Sep 09 20:26:51.085272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085241    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kube-rbac-proxy"
Sep 09 20:26:51.085272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085250    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5"
Sep 09 20:26:51.085272 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085256    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5"
Sep 09 20:26:51.085476 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085303    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kube-rbac-proxy"
Sep 09 20:26:51.085476 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085313    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c2112d-2c84-404e-a697-caf13b8d9195" containerName="kserve-container"
Sep 09 20:26:51.085476 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.085322    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6605a9e0-11e4-43e4-85ce-77d0ae9bba6d" containerName="switch-graph-510d5"
Sep 09 20:26:51.088716 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.088699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.091220 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.091196    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 09 20:26:51.091306 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.091240    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 09 20:26:51.095659 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.095627    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"]
Sep 09 20:26:51.200363 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.200339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bfcad6-0845-47c0-aac0-f4eb30c69102-openshift-service-ca-bundle\") pod \"model-chainer-6f496df4c6-8w6hl\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") " pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.200480 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.200367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8bfcad6-0845-47c0-aac0-f4eb30c69102-proxy-tls\") pod \"model-chainer-6f496df4c6-8w6hl\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") " pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.301110 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.301043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bfcad6-0845-47c0-aac0-f4eb30c69102-openshift-service-ca-bundle\") pod \"model-chainer-6f496df4c6-8w6hl\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") " pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.301110 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.301073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8bfcad6-0845-47c0-aac0-f4eb30c69102-proxy-tls\") pod \"model-chainer-6f496df4c6-8w6hl\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") " pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.301618 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.301591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bfcad6-0845-47c0-aac0-f4eb30c69102-openshift-service-ca-bundle\") pod \"model-chainer-6f496df4c6-8w6hl\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") " pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.303357 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.303332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8bfcad6-0845-47c0-aac0-f4eb30c69102-proxy-tls\") pod \"model-chainer-6f496df4c6-8w6hl\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") " pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.399827 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.399808    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:51.665432 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.665261    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"]
Sep 09 20:26:51.668127 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:26:51.668099    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bfcad6_0845_47c0_aac0_f4eb30c69102.slice/crio-157654b2d97e6b3032b3a02d02a2e322ba6fae1f78caf96078d753e29b0b5393 WatchSource:0}: Error finding container 157654b2d97e6b3032b3a02d02a2e322ba6fae1f78caf96078d753e29b0b5393: Status 404 returned error can't find the container with id 157654b2d97e6b3032b3a02d02a2e322ba6fae1f78caf96078d753e29b0b5393
Sep 09 20:26:51.997931 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.997843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" event={"ID":"b8bfcad6-0845-47c0-aac0-f4eb30c69102","Type":"ContainerStarted","Data":"351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061"}
Sep 09 20:26:51.997931 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.997892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" event={"ID":"b8bfcad6-0845-47c0-aac0-f4eb30c69102","Type":"ContainerStarted","Data":"157654b2d97e6b3032b3a02d02a2e322ba6fae1f78caf96078d753e29b0b5393"}
Sep 09 20:26:51.998163 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:51.997950    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:52.014747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:52.014694    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podStartSLOduration=1.014680625 podStartE2EDuration="1.014680625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:52.013792113 +0000 UTC m=+927.987562199" watchObservedRunningTime="2026-09-09 20:26:52.014680625 +0000 UTC m=+927.988450687"
Sep 09 20:26:58.008097 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:58.008068    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:26:59.835163 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:26:59.835132    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:27:01.191993 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.191961    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"]
Sep 09 20:27:01.192465 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.192172    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer" containerID="cri-o://351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061" gracePeriod=30
Sep 09 20:27:01.370790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.370756    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"]
Sep 09 20:27:01.371124 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.371074    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container" containerID="cri-o://3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16" gracePeriod=30
Sep 09 20:27:01.371214 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.371125    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kube-rbac-proxy" containerID="cri-o://50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15" gracePeriod=30
Sep 09 20:27:01.456352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.456279    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"]
Sep 09 20:27:01.459774 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.459759    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.462273 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.462256    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53cb1-predictor-serving-cert\""
Sep 09 20:27:01.462336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.462277    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53cb1-kube-rbac-proxy-sar-config\""
Sep 09 20:27:01.470449 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.470423    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"]
Sep 09 20:27:01.581940 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.581908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.581940 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.581940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-api-access-fgpv5\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.582146 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.581962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.682586 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.682548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.682586 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.682586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-api-access-fgpv5\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.682807 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.682611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.682807 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:27:01.682717    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-serving-cert: secret "success-200-isvc-53cb1-predictor-serving-cert" not found
Sep 09 20:27:01.682807 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:27:01.682806    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls podName:32b28779-e0dd-4d7a-be0f-8216b7f86752 nodeName:}" failed. No retries permitted until 2026-09-09 20:27:02.18278503 +0000 UTC m=+938.156555073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls") pod "success-200-isvc-53cb1-predictor-77bd468459-7gnsg" (UID: "32b28779-e0dd-4d7a-be0f-8216b7f86752") : secret "success-200-isvc-53cb1-predictor-serving-cert" not found
Sep 09 20:27:01.683294 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.683277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:01.694490 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:01.694469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-api-access-fgpv5\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:02.043347 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:02.043311    2562 generic.go:356] "Generic (PLEG): container finished" podID="53132e26-85c8-4266-a005-b79dec2091c9" containerID="50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15" exitCode=2
Sep 09 20:27:02.043511 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:02.043384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerDied","Data":"50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15"}
Sep 09 20:27:02.187742 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:02.187705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:02.190129 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:02.190099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls\") pod \"success-200-isvc-53cb1-predictor-77bd468459-7gnsg\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") " pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:02.369819 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:02.369726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:02.639013 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:02.638980    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"]
Sep 09 20:27:02.640663 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:27:02.640632    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b28779_e0dd_4d7a_be0f_8216b7f86752.slice/crio-d9b0b584cc1758b8a183958d06d67e2328f8e62a413b8f8712797fa0b128129c WatchSource:0}: Error finding container d9b0b584cc1758b8a183958d06d67e2328f8e62a413b8f8712797fa0b128129c: Status 404 returned error can't find the container with id d9b0b584cc1758b8a183958d06d67e2328f8e62a413b8f8712797fa0b128129c
Sep 09 20:27:03.006415 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:03.006318    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:03.050226 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:03.050191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" event={"ID":"32b28779-e0dd-4d7a-be0f-8216b7f86752","Type":"ContainerStarted","Data":"e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6"}
Sep 09 20:27:03.050386 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:03.050258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" event={"ID":"32b28779-e0dd-4d7a-be0f-8216b7f86752","Type":"ContainerStarted","Data":"34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb"}
Sep 09 20:27:03.050386 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:03.050284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" event={"ID":"32b28779-e0dd-4d7a-be0f-8216b7f86752","Type":"ContainerStarted","Data":"d9b0b584cc1758b8a183958d06d67e2328f8e62a413b8f8712797fa0b128129c"}
Sep 09 20:27:03.050386 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:03.050314    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:03.068256 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:03.068204    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podStartSLOduration=2.068188467 podStartE2EDuration="2.068188467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:03.066266114 +0000 UTC m=+939.040036188" watchObservedRunningTime="2026-09-09 20:27:03.068188467 +0000 UTC m=+939.041958530"
Sep 09 20:27:04.055645 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:04.055610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:04.056881 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:04.056856    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 20:27:05.023576 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.023545    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:27:05.062025 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.061967    2562 generic.go:356] "Generic (PLEG): container finished" podID="53132e26-85c8-4266-a005-b79dec2091c9" containerID="3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16" exitCode=0
Sep 09 20:27:05.062434 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.062031    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"
Sep 09 20:27:05.062434 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.062044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerDied","Data":"3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16"}
Sep 09 20:27:05.062434 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.062112    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf" event={"ID":"53132e26-85c8-4266-a005-b79dec2091c9","Type":"ContainerDied","Data":"faeca9a3f412630196e4218ea3a8e5087cc465345c2d6229580d99435aaac591"}
Sep 09 20:27:05.062434 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.062149    2562 scope.go:117] "RemoveContainer" containerID="50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15"
Sep 09 20:27:05.062636 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.062570    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 20:27:05.071275 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.071260    2562 scope.go:117] "RemoveContainer" containerID="3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16"
Sep 09 20:27:05.078917 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.078888    2562 scope.go:117] "RemoveContainer" containerID="bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd"
Sep 09 20:27:05.086198 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.086178    2562 scope.go:117] "RemoveContainer" containerID="50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15"
Sep 09 20:27:05.086474 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:27:05.086452    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15\": container with ID starting with 50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15 not found: ID does not exist" containerID="50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15"
Sep 09 20:27:05.086543 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.086488    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15"} err="failed to get container status \"50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15\": rpc error: code = NotFound desc = could not find container \"50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15\": container with ID starting with 50a50a6b52d80de87e6ff416480957cdf07ec59bf6af57d1b802e0c0381c9f15 not found: ID does not exist"
Sep 09 20:27:05.086543 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.086514    2562 scope.go:117] "RemoveContainer" containerID="3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16"
Sep 09 20:27:05.086784 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:27:05.086767    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16\": container with ID starting with 3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16 not found: ID does not exist" containerID="3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16"
Sep 09 20:27:05.086828 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.086790    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16"} err="failed to get container status \"3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16\": rpc error: code = NotFound desc = could not find container \"3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16\": container with ID starting with 3e0083d32008d96c6f5ee867a022c1044cf6135e4918f5356b618401d4475c16 not found: ID does not exist"
Sep 09 20:27:05.086828 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.086804    2562 scope.go:117] "RemoveContainer" containerID="bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd"
Sep 09 20:27:05.087076 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:27:05.087058    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd\": container with ID starting with bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd not found: ID does not exist" containerID="bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd"
Sep 09 20:27:05.087116 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.087080    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd"} err="failed to get container status \"bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd\": rpc error: code = NotFound desc = could not find container \"bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd\": container with ID starting with bdec6683192b8026daf6563917f40e63ae2bc9d622c8c4f430afe0a3bf9dafcd not found: ID does not exist"
Sep 09 20:27:05.110407 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.110376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls\") pod \"53132e26-85c8-4266-a005-b79dec2091c9\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") "
Sep 09 20:27:05.110549 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.110419    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-kube-rbac-proxy-sar-config\") pod \"53132e26-85c8-4266-a005-b79dec2091c9\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") "
Sep 09 20:27:05.110549 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.110460    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p9q5\" (UniqueName: \"kubernetes.io/projected/53132e26-85c8-4266-a005-b79dec2091c9-kube-api-access-6p9q5\") pod \"53132e26-85c8-4266-a005-b79dec2091c9\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") "
Sep 09 20:27:05.110549 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.110534    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53132e26-85c8-4266-a005-b79dec2091c9-kserve-provision-location\") pod \"53132e26-85c8-4266-a005-b79dec2091c9\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") "
Sep 09 20:27:05.110702 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.110561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-cabundle-cert\") pod \"53132e26-85c8-4266-a005-b79dec2091c9\" (UID: \"53132e26-85c8-4266-a005-b79dec2091c9\") "
Sep 09 20:27:05.110938 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.110906    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53132e26-85c8-4266-a005-b79dec2091c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53132e26-85c8-4266-a005-b79dec2091c9" (UID: "53132e26-85c8-4266-a005-b79dec2091c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:05.111204 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.111179    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53132e26-85c8-4266-a005-b79dec2091c9" (UID: "53132e26-85c8-4266-a005-b79dec2091c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:05.111334 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.111305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53132e26-85c8-4266-a005-b79dec2091c9" (UID: "53132e26-85c8-4266-a005-b79dec2091c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:05.112524 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.112503    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53132e26-85c8-4266-a005-b79dec2091c9-kube-api-access-6p9q5" (OuterVolumeSpecName: "kube-api-access-6p9q5") pod "53132e26-85c8-4266-a005-b79dec2091c9" (UID: "53132e26-85c8-4266-a005-b79dec2091c9"). InnerVolumeSpecName "kube-api-access-6p9q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:05.112607 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.112593    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53132e26-85c8-4266-a005-b79dec2091c9" (UID: "53132e26-85c8-4266-a005-b79dec2091c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:05.211268 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.211234    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p9q5\" (UniqueName: \"kubernetes.io/projected/53132e26-85c8-4266-a005-b79dec2091c9-kube-api-access-6p9q5\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:27:05.211268 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.211263    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53132e26-85c8-4266-a005-b79dec2091c9-kserve-provision-location\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:27:05.211268 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.211272    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-cabundle-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:27:05.211486 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.211282    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53132e26-85c8-4266-a005-b79dec2091c9-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:27:05.211486 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.211290    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53132e26-85c8-4266-a005-b79dec2091c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:27:05.383520 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.383487    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"]
Sep 09 20:27:05.387628 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:05.387605    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ksclf"]
Sep 09 20:27:06.464959 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:06.464926    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53132e26-85c8-4266-a005-b79dec2091c9" path="/var/lib/kubelet/pods/53132e26-85c8-4266-a005-b79dec2091c9/volumes"
Sep 09 20:27:08.005487 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:08.005449    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:10.067091 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:10.067058    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:27:10.067573 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:10.067543    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 20:27:11.492268 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492237    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"]
Sep 09 20:27:11.492611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492561    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kube-rbac-proxy"
Sep 09 20:27:11.492611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492572    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kube-rbac-proxy"
Sep 09 20:27:11.492611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492582    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="storage-initializer"
Sep 09 20:27:11.492611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492587    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="storage-initializer"
Sep 09 20:27:11.492611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492602    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container"
Sep 09 20:27:11.492611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492608    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container"
Sep 09 20:27:11.492799 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492659    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kube-rbac-proxy"
Sep 09 20:27:11.492799 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.492667    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53132e26-85c8-4266-a005-b79dec2091c9" containerName="kserve-container"
Sep 09 20:27:11.497139 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.497120    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:11.499666 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.499644    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-2e273-kube-rbac-proxy-sar-config\""
Sep 09 20:27:11.499767 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.499701    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-2e273-serving-cert\""
Sep 09 20:27:11.503612 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.503586    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"]
Sep 09 20:27:11.564872 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.564843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/076cd625-1e91-41fe-b35b-6a6ac7c7a756-openshift-service-ca-bundle\") pod \"switch-graph-2e273-55d76c96b5-pbttp\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") " pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:11.565041 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.564905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/076cd625-1e91-41fe-b35b-6a6ac7c7a756-proxy-tls\") pod \"switch-graph-2e273-55d76c96b5-pbttp\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") " pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:11.665965 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.665935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/076cd625-1e91-41fe-b35b-6a6ac7c7a756-openshift-service-ca-bundle\") pod \"switch-graph-2e273-55d76c96b5-pbttp\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") " pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:11.666193 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.665986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/076cd625-1e91-41fe-b35b-6a6ac7c7a756-proxy-tls\") pod \"switch-graph-2e273-55d76c96b5-pbttp\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") " pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:11.666620 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.666594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/076cd625-1e91-41fe-b35b-6a6ac7c7a756-openshift-service-ca-bundle\") pod \"switch-graph-2e273-55d76c96b5-pbttp\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") " pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:11.668175 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.668159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/076cd625-1e91-41fe-b35b-6a6ac7c7a756-proxy-tls\") pod \"switch-graph-2e273-55d76c96b5-pbttp\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") " pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:11.808955 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:11.808888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:12.077103 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:12.077029    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"]
Sep 09 20:27:12.081182 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:27:12.081143    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076cd625_1e91_41fe_b35b_6a6ac7c7a756.slice/crio-05c0bc588a27dcf4ac84b9aa295b3fe87f09db45c83fd1846e9a57070924c963 WatchSource:0}: Error finding container 05c0bc588a27dcf4ac84b9aa295b3fe87f09db45c83fd1846e9a57070924c963: Status 404 returned error can't find the container with id 05c0bc588a27dcf4ac84b9aa295b3fe87f09db45c83fd1846e9a57070924c963
Sep 09 20:27:12.097515 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:12.097484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" event={"ID":"076cd625-1e91-41fe-b35b-6a6ac7c7a756","Type":"ContainerStarted","Data":"05c0bc588a27dcf4ac84b9aa295b3fe87f09db45c83fd1846e9a57070924c963"}
Sep 09 20:27:13.005780 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:13.005739    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:13.006150 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:13.005837    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:27:13.105294 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:13.105264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" event={"ID":"076cd625-1e91-41fe-b35b-6a6ac7c7a756","Type":"ContainerStarted","Data":"177892835b9ffdd12cb2fa3ab14c1d388bf8b0c03bac928c99813fca34b6e6ac"}
Sep 09 20:27:13.105448 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:13.105324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:13.122508 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:13.122465    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podStartSLOduration=2.122451108 podStartE2EDuration="2.122451108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:13.122019709 +0000 UTC m=+949.095789766" watchObservedRunningTime="2026-09-09 20:27:13.122451108 +0000 UTC m=+949.096221170"
Sep 09 20:27:18.006100 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:18.006055    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:19.115851 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:19.115824    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:27:20.068511 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:20.068475    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 20:27:23.006018 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:23.005958    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:28.018783 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:28.018729    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:30.068146 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:30.068106    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 20:27:31.839327 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:31.839301    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:27:31.926290 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:31.926253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8bfcad6-0845-47c0-aac0-f4eb30c69102-proxy-tls\") pod \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") "
Sep 09 20:27:31.926487 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:31.926320    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bfcad6-0845-47c0-aac0-f4eb30c69102-openshift-service-ca-bundle\") pod \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\" (UID: \"b8bfcad6-0845-47c0-aac0-f4eb30c69102\") "
Sep 09 20:27:31.926685 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:31.926658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bfcad6-0845-47c0-aac0-f4eb30c69102-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b8bfcad6-0845-47c0-aac0-f4eb30c69102" (UID: "b8bfcad6-0845-47c0-aac0-f4eb30c69102"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:31.928356 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:31.928336    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8bfcad6-0845-47c0-aac0-f4eb30c69102-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8bfcad6-0845-47c0-aac0-f4eb30c69102" (UID: "b8bfcad6-0845-47c0-aac0-f4eb30c69102"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:32.027057 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.026949    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8bfcad6-0845-47c0-aac0-f4eb30c69102-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:27:32.027057 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.026983    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bfcad6-0845-47c0-aac0-f4eb30c69102-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:27:32.192286 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.192252    2562 generic.go:356] "Generic (PLEG): container finished" podID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerID="351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061" exitCode=137
Sep 09 20:27:32.192441 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.192298    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"
Sep 09 20:27:32.192441 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.192338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" event={"ID":"b8bfcad6-0845-47c0-aac0-f4eb30c69102","Type":"ContainerDied","Data":"351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061"}
Sep 09 20:27:32.192441 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.192388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl" event={"ID":"b8bfcad6-0845-47c0-aac0-f4eb30c69102","Type":"ContainerDied","Data":"157654b2d97e6b3032b3a02d02a2e322ba6fae1f78caf96078d753e29b0b5393"}
Sep 09 20:27:32.192441 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.192409    2562 scope.go:117] "RemoveContainer" containerID="351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061"
Sep 09 20:27:32.200969 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.200737    2562 scope.go:117] "RemoveContainer" containerID="351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061"
Sep 09 20:27:32.200969 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:27:32.200960    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061\": container with ID starting with 351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061 not found: ID does not exist" containerID="351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061"
Sep 09 20:27:32.201090 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.200986    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061"} err="failed to get container status \"351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061\": rpc error: code = NotFound desc = could not find container \"351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061\": container with ID starting with 351d0825590496f9b622b80206ca38bb550f670293e807ef912abaca46ebe061 not found: ID does not exist"
Sep 09 20:27:32.212497 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.212468    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"]
Sep 09 20:27:32.214307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.214285    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f496df4c6-8w6hl"]
Sep 09 20:27:32.465213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:32.465170    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" path="/var/lib/kubelet/pods/b8bfcad6-0845-47c0-aac0-f4eb30c69102/volumes"
Sep 09 20:27:40.067846 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:40.067801    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 20:27:50.068760 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:27:50.068731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:28:01.396260 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.396221    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"]
Sep 09 20:28:01.396617 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.396574    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer"
Sep 09 20:28:01.396617 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.396586    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer"
Sep 09 20:28:01.396686 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.396650    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8bfcad6-0845-47c0-aac0-f4eb30c69102" containerName="model-chainer"
Sep 09 20:28:01.399376 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.399359    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.401774 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.401755    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-53cb1-serving-cert\""
Sep 09 20:28:01.401908 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.401890    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-53cb1-kube-rbac-proxy-sar-config\""
Sep 09 20:28:01.409529 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.409506    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"]
Sep 09 20:28:01.456196 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.456163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c756ac-a262-48aa-9467-34510e4b9602-proxy-tls\") pod \"sequence-graph-53cb1-d889798ff-6bkzb\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") " pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.456344 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.456262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c756ac-a262-48aa-9467-34510e4b9602-openshift-service-ca-bundle\") pod \"sequence-graph-53cb1-d889798ff-6bkzb\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") " pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.556788 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.556748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c756ac-a262-48aa-9467-34510e4b9602-proxy-tls\") pod \"sequence-graph-53cb1-d889798ff-6bkzb\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") " pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.556955 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.556854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c756ac-a262-48aa-9467-34510e4b9602-openshift-service-ca-bundle\") pod \"sequence-graph-53cb1-d889798ff-6bkzb\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") " pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.557603 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.557582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c756ac-a262-48aa-9467-34510e4b9602-openshift-service-ca-bundle\") pod \"sequence-graph-53cb1-d889798ff-6bkzb\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") " pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.559241 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.559222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c756ac-a262-48aa-9467-34510e4b9602-proxy-tls\") pod \"sequence-graph-53cb1-d889798ff-6bkzb\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") " pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.709218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.709125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:01.979064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:01.979039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"]
Sep 09 20:28:01.980821 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:28:01.980795    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c756ac_a262_48aa_9467_34510e4b9602.slice/crio-9dea81a423b7154ccef908b7d61d2e06e11f55da2182e43b67414d830159e1dd WatchSource:0}: Error finding container 9dea81a423b7154ccef908b7d61d2e06e11f55da2182e43b67414d830159e1dd: Status 404 returned error can't find the container with id 9dea81a423b7154ccef908b7d61d2e06e11f55da2182e43b67414d830159e1dd
Sep 09 20:28:02.326330 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:02.326234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" event={"ID":"56c756ac-a262-48aa-9467-34510e4b9602","Type":"ContainerStarted","Data":"b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059"}
Sep 09 20:28:02.326527 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:02.326390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:28:02.326527 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:02.326414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" event={"ID":"56c756ac-a262-48aa-9467-34510e4b9602","Type":"ContainerStarted","Data":"9dea81a423b7154ccef908b7d61d2e06e11f55da2182e43b67414d830159e1dd"}
Sep 09 20:28:02.343098 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:02.343048    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podStartSLOduration=1.343032673 podStartE2EDuration="1.343032673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:02.341786652 +0000 UTC m=+998.315556714" watchObservedRunningTime="2026-09-09 20:28:02.343032673 +0000 UTC m=+998.316802736"
Sep 09 20:28:08.337415 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:28:08.337385    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:31:24.463236 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:31:24.463200    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:31:24.465530 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:31:24.465509    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:35:26.075315 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.075218    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"]
Sep 09 20:35:26.075818 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.075461    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273" containerID="cri-o://177892835b9ffdd12cb2fa3ab14c1d388bf8b0c03bac928c99813fca34b6e6ac" gracePeriod=30
Sep 09 20:35:26.197114 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.197081    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"]
Sep 09 20:35:26.274676 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.274636    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container" containerID="cri-o://98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79" gracePeriod=30
Sep 09 20:35:26.274885 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.274671    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kube-rbac-proxy" containerID="cri-o://ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71" gracePeriod=30
Sep 09 20:35:26.280928 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.280900    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"]
Sep 09 20:35:26.284387 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.284367    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.286951 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.286928    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b3d7b-predictor-serving-cert\""
Sep 09 20:35:26.287077 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.286929    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b3d7b-kube-rbac-proxy-sar-config\""
Sep 09 20:35:26.294036 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.294010    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"]
Sep 09 20:35:26.385310 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.385271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stmx\" (UniqueName: \"kubernetes.io/projected/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-api-access-2stmx\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.385310 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.385315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.385548 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.385434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.486448 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.486409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.486613 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.486478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stmx\" (UniqueName: \"kubernetes.io/projected/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-api-access-2stmx\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.486613 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.486499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.486757 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:35:26.486624    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-serving-cert: secret "success-200-isvc-b3d7b-predictor-serving-cert" not found
Sep 09 20:35:26.486757 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:35:26.486682    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls podName:142fe19b-06bd-4efa-a0a3-bb06d998517d nodeName:}" failed. No retries permitted until 2026-09-09 20:35:26.986663152 +0000 UTC m=+1442.960433196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls") pod "success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" (UID: "142fe19b-06bd-4efa-a0a3-bb06d998517d") : secret "success-200-isvc-b3d7b-predictor-serving-cert" not found
Sep 09 20:35:26.487056 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.487037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.496340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.496308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stmx\" (UniqueName: \"kubernetes.io/projected/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-api-access-2stmx\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.990856 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.990816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:26.993248 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:26.993220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls\") pod \"success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") " pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:27.200892 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:27.200852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:27.281897 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:27.281859    2562 generic.go:356] "Generic (PLEG): container finished" podID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerID="ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71" exitCode=2
Sep 09 20:35:27.282071 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:27.281925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" event={"ID":"9ea26b1b-7215-49c1-8a18-5c91d850e6bd","Type":"ContainerDied","Data":"ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71"}
Sep 09 20:35:27.472231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:27.472208    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"]
Sep 09 20:35:27.474518 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:35:27.474491    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142fe19b_06bd_4efa_a0a3_bb06d998517d.slice/crio-b11085877e243452aca94f7bed19f208dddcef7127898638c5876290241ef6c6 WatchSource:0}: Error finding container b11085877e243452aca94f7bed19f208dddcef7127898638c5876290241ef6c6: Status 404 returned error can't find the container with id b11085877e243452aca94f7bed19f208dddcef7127898638c5876290241ef6c6
Sep 09 20:35:28.288779 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:28.288738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" event={"ID":"142fe19b-06bd-4efa-a0a3-bb06d998517d","Type":"ContainerStarted","Data":"cbd1f27f8595782fbb593254653febb9f206ce3b81d98043b9af9bf472b77951"}
Sep 09 20:35:28.289199 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:28.288808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" event={"ID":"142fe19b-06bd-4efa-a0a3-bb06d998517d","Type":"ContainerStarted","Data":"f554ebc0028f6310de2f0c27c23a56f4141ada1fefb4c07307025609da220153"}
Sep 09 20:35:28.289199 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:28.288838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" event={"ID":"142fe19b-06bd-4efa-a0a3-bb06d998517d","Type":"ContainerStarted","Data":"b11085877e243452aca94f7bed19f208dddcef7127898638c5876290241ef6c6"}
Sep 09 20:35:28.289199 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:28.288975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:28.307229 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:28.307173    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podStartSLOduration=2.307159337 podStartE2EDuration="2.307159337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:28.305956981 +0000 UTC m=+1444.279727043" watchObservedRunningTime="2026-09-09 20:35:28.307159337 +0000 UTC m=+1444.280929400"
Sep 09 20:35:29.115644 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.115597    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:29.293760 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.293728    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:29.295133 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.295103    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:35:29.638123 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.638096    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:35:29.726153 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.726063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-rbac-proxy-sar-config\") pod \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") "
Sep 09 20:35:29.726153 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.726100    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snh7x\" (UniqueName: \"kubernetes.io/projected/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-api-access-snh7x\") pod \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") "
Sep 09 20:35:29.726153 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.726140    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-proxy-tls\") pod \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\" (UID: \"9ea26b1b-7215-49c1-8a18-5c91d850e6bd\") "
Sep 09 20:35:29.726477 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.726453    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ea26b1b-7215-49c1-8a18-5c91d850e6bd" (UID: "9ea26b1b-7215-49c1-8a18-5c91d850e6bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:29.728180 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.728155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-api-access-snh7x" (OuterVolumeSpecName: "kube-api-access-snh7x") pod "9ea26b1b-7215-49c1-8a18-5c91d850e6bd" (UID: "9ea26b1b-7215-49c1-8a18-5c91d850e6bd"). InnerVolumeSpecName "kube-api-access-snh7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:29.728313 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.728290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ea26b1b-7215-49c1-8a18-5c91d850e6bd" (UID: "9ea26b1b-7215-49c1-8a18-5c91d850e6bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:29.827017 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.826972    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:35:29.827180 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.827029    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snh7x\" (UniqueName: \"kubernetes.io/projected/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-kube-api-access-snh7x\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:35:29.827180 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:29.827046    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea26b1b-7215-49c1-8a18-5c91d850e6bd-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:35:30.299204 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.299170    2562 generic.go:356] "Generic (PLEG): container finished" podID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerID="98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79" exitCode=0
Sep 09 20:35:30.299644 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.299222    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"
Sep 09 20:35:30.299644 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.299262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" event={"ID":"9ea26b1b-7215-49c1-8a18-5c91d850e6bd","Type":"ContainerDied","Data":"98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79"}
Sep 09 20:35:30.299644 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.299315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9" event={"ID":"9ea26b1b-7215-49c1-8a18-5c91d850e6bd","Type":"ContainerDied","Data":"b7c2e13dd8299f05412d786dd48e5fafa02f8007f97de8c8b139e0f69c8a4350"}
Sep 09 20:35:30.299644 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.299341    2562 scope.go:117] "RemoveContainer" containerID="ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71"
Sep 09 20:35:30.300017 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.299973    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:35:30.308716 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.308698    2562 scope.go:117] "RemoveContainer" containerID="98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79"
Sep 09 20:35:30.316389 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.316372    2562 scope.go:117] "RemoveContainer" containerID="ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71"
Sep 09 20:35:30.316674 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:35:30.316653    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71\": container with ID starting with ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71 not found: ID does not exist" containerID="ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71"
Sep 09 20:35:30.316721 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.316681    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71"} err="failed to get container status \"ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71\": rpc error: code = NotFound desc = could not find container \"ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71\": container with ID starting with ef85b93d8c2053bad0a89572e2189c0deb2668c236e8dc03886e9fb5f4da4b71 not found: ID does not exist"
Sep 09 20:35:30.316721 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.316698    2562 scope.go:117] "RemoveContainer" containerID="98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79"
Sep 09 20:35:30.316937 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:35:30.316918    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79\": container with ID starting with 98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79 not found: ID does not exist" containerID="98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79"
Sep 09 20:35:30.316995 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.316946    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79"} err="failed to get container status \"98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79\": rpc error: code = NotFound desc = could not find container \"98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79\": container with ID starting with 98059c1ff990fe4881daf78b13b3a5cb7e6b81be9ceb84b0fdd48cb2e52cbb79 not found: ID does not exist"
Sep 09 20:35:30.320376 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.320352    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"]
Sep 09 20:35:30.321828 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.321806    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e273-predictor-85c84c9849-pcfd9"]
Sep 09 20:35:30.465323 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:30.465292    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" path="/var/lib/kubelet/pods/9ea26b1b-7215-49c1-8a18-5c91d850e6bd/volumes"
Sep 09 20:35:34.114271 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:34.114227    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:35.303991 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:35.303962    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:35:35.304490 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:35.304463    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:35:39.114731 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:39.114670    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:39.115145 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:39.114852    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:35:44.114508 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:44.114470    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:45.305185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:45.305144    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:35:49.114055 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:49.113982    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:54.113676 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:54.113632    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:55.305379 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:55.305338    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:35:56.415748 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.415717    2562 generic.go:356] "Generic (PLEG): container finished" podID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerID="177892835b9ffdd12cb2fa3ab14c1d388bf8b0c03bac928c99813fca34b6e6ac" exitCode=0
Sep 09 20:35:56.416106 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.415801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" event={"ID":"076cd625-1e91-41fe-b35b-6a6ac7c7a756","Type":"ContainerDied","Data":"177892835b9ffdd12cb2fa3ab14c1d388bf8b0c03bac928c99813fca34b6e6ac"}
Sep 09 20:35:56.719796 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.719774    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:35:56.734413 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.734389    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/076cd625-1e91-41fe-b35b-6a6ac7c7a756-proxy-tls\") pod \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") "
Sep 09 20:35:56.734521 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.734426    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/076cd625-1e91-41fe-b35b-6a6ac7c7a756-openshift-service-ca-bundle\") pod \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\" (UID: \"076cd625-1e91-41fe-b35b-6a6ac7c7a756\") "
Sep 09 20:35:56.734819 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.734764    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076cd625-1e91-41fe-b35b-6a6ac7c7a756-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "076cd625-1e91-41fe-b35b-6a6ac7c7a756" (UID: "076cd625-1e91-41fe-b35b-6a6ac7c7a756"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:56.736685 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.736660    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076cd625-1e91-41fe-b35b-6a6ac7c7a756-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "076cd625-1e91-41fe-b35b-6a6ac7c7a756" (UID: "076cd625-1e91-41fe-b35b-6a6ac7c7a756"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:56.834948 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.834912    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/076cd625-1e91-41fe-b35b-6a6ac7c7a756-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:35:56.834948 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:56.834946    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/076cd625-1e91-41fe-b35b-6a6ac7c7a756-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:35:57.421609 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:57.421582    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"
Sep 09 20:35:57.421609 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:57.421599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp" event={"ID":"076cd625-1e91-41fe-b35b-6a6ac7c7a756","Type":"ContainerDied","Data":"05c0bc588a27dcf4ac84b9aa295b3fe87f09db45c83fd1846e9a57070924c963"}
Sep 09 20:35:57.422154 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:57.421651    2562 scope.go:117] "RemoveContainer" containerID="177892835b9ffdd12cb2fa3ab14c1d388bf8b0c03bac928c99813fca34b6e6ac"
Sep 09 20:35:57.442739 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:57.442712    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"]
Sep 09 20:35:57.446580 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:57.446549    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2e273-55d76c96b5-pbttp"]
Sep 09 20:35:58.465259 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:35:58.465229    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" path="/var/lib/kubelet/pods/076cd625-1e91-41fe-b35b-6a6ac7c7a756/volumes"
Sep 09 20:36:05.304583 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:05.304542    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:36:15.305201 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:15.305163    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:36:16.079807 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.079776    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"]
Sep 09 20:36:16.080031 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.079970    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1" containerID="cri-o://b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059" gracePeriod=30
Sep 09 20:36:16.169509 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.169473    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"]
Sep 09 20:36:16.169743 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.169719    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container" containerID="cri-o://34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb" gracePeriod=30
Sep 09 20:36:16.169824 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.169742    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kube-rbac-proxy" containerID="cri-o://e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6" gracePeriod=30
Sep 09 20:36:16.288808 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.288779    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"]
Sep 09 20:36:16.289151 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289137    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273"
Sep 09 20:36:16.289200 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289153    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273"
Sep 09 20:36:16.289200 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289164    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container"
Sep 09 20:36:16.289200 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289170    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container"
Sep 09 20:36:16.289200 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289179    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kube-rbac-proxy"
Sep 09 20:36:16.289200 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289184    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kube-rbac-proxy"
Sep 09 20:36:16.289367 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289239    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kube-rbac-proxy"
Sep 09 20:36:16.289367 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289249    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="076cd625-1e91-41fe-b35b-6a6ac7c7a756" containerName="switch-graph-2e273"
Sep 09 20:36:16.289367 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.289255    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea26b1b-7215-49c1-8a18-5c91d850e6bd" containerName="kserve-container"
Sep 09 20:36:16.292105 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.292089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.294528 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.294508    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-90077-predictor-serving-cert\""
Sep 09 20:36:16.294628 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.294511    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-90077-kube-rbac-proxy-sar-config\""
Sep 09 20:36:16.302806 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.302783    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"]
Sep 09 20:36:16.398336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.398306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.398656 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.398358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.398656 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.398390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jl4\" (UniqueName: \"kubernetes.io/projected/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-api-access-s8jl4\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.499033 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.498986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.499164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.499126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.499164 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.499154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jl4\" (UniqueName: \"kubernetes.io/projected/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-api-access-s8jl4\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.499257 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:36:16.499239    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-90077-predictor-serving-cert: secret "success-200-isvc-90077-predictor-serving-cert" not found
Sep 09 20:36:16.499308 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:36:16.499298    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls podName:d1d2a4bd-d2f8-489c-a578-59501bda5835 nodeName:}" failed. No retries permitted until 2026-09-09 20:36:16.999282015 +0000 UTC m=+1492.973052056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls") pod "success-200-isvc-90077-predictor-58445bbccc-n7mjk" (UID: "d1d2a4bd-d2f8-489c-a578-59501bda5835") : secret "success-200-isvc-90077-predictor-serving-cert" not found
Sep 09 20:36:16.499636 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.499619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:16.506845 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.506817    2562 generic.go:356] "Generic (PLEG): container finished" podID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerID="e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6" exitCode=2
Sep 09 20:36:16.506957 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.506851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" event={"ID":"32b28779-e0dd-4d7a-be0f-8216b7f86752","Type":"ContainerDied","Data":"e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6"}
Sep 09 20:36:16.507742 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:16.507715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jl4\" (UniqueName: \"kubernetes.io/projected/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-api-access-s8jl4\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:17.003268 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:17.003221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:17.005527 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:17.005507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls\") pod \"success-200-isvc-90077-predictor-58445bbccc-n7mjk\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") " pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:17.202310 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:17.202281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:17.474071 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:17.474044    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"]
Sep 09 20:36:17.475853 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:36:17.475817    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d2a4bd_d2f8_489c_a578_59501bda5835.slice/crio-cf9f6ec40567ca8df7af908f48bf75d5c17070349beee0052837ab978bf9a9e5 WatchSource:0}: Error finding container cf9f6ec40567ca8df7af908f48bf75d5c17070349beee0052837ab978bf9a9e5: Status 404 returned error can't find the container with id cf9f6ec40567ca8df7af908f48bf75d5c17070349beee0052837ab978bf9a9e5
Sep 09 20:36:17.512600 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:17.512576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" event={"ID":"d1d2a4bd-d2f8-489c-a578-59501bda5835","Type":"ContainerStarted","Data":"cf9f6ec40567ca8df7af908f48bf75d5c17070349beee0052837ab978bf9a9e5"}
Sep 09 20:36:18.335621 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:18.335580    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:18.519820 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:18.519785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" event={"ID":"d1d2a4bd-d2f8-489c-a578-59501bda5835","Type":"ContainerStarted","Data":"1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59"}
Sep 09 20:36:18.520205 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:18.519840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" event={"ID":"d1d2a4bd-d2f8-489c-a578-59501bda5835","Type":"ContainerStarted","Data":"24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa"}
Sep 09 20:36:18.520205 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:18.520020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:18.520205 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:18.520071    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:18.521298 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:18.521274    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:36:18.536432 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:18.536382    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podStartSLOduration=2.536367177 podStartE2EDuration="2.536367177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:18.53498517 +0000 UTC m=+1494.508755233" watchObservedRunningTime="2026-09-09 20:36:18.536367177 +0000 UTC m=+1494.510137240"
Sep 09 20:36:19.118875 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.118852    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:36:19.223175 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.223109    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls\") pod \"32b28779-e0dd-4d7a-be0f-8216b7f86752\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") "
Sep 09 20:36:19.223175 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.223150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-rbac-proxy-sar-config\") pod \"32b28779-e0dd-4d7a-be0f-8216b7f86752\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") "
Sep 09 20:36:19.223383 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.223187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-api-access-fgpv5\") pod \"32b28779-e0dd-4d7a-be0f-8216b7f86752\" (UID: \"32b28779-e0dd-4d7a-be0f-8216b7f86752\") "
Sep 09 20:36:19.223508 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.223486    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32b28779-e0dd-4d7a-be0f-8216b7f86752" (UID: "32b28779-e0dd-4d7a-be0f-8216b7f86752"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:19.225190 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.225170    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32b28779-e0dd-4d7a-be0f-8216b7f86752" (UID: "32b28779-e0dd-4d7a-be0f-8216b7f86752"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:19.225264 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.225225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-api-access-fgpv5" (OuterVolumeSpecName: "kube-api-access-fgpv5") pod "32b28779-e0dd-4d7a-be0f-8216b7f86752" (UID: "32b28779-e0dd-4d7a-be0f-8216b7f86752"). InnerVolumeSpecName "kube-api-access-fgpv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:19.324262 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.324236    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b28779-e0dd-4d7a-be0f-8216b7f86752-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:19.324262 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.324258    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:19.324262 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.324267    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/32b28779-e0dd-4d7a-be0f-8216b7f86752-kube-api-access-fgpv5\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:19.525338 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.525253    2562 generic.go:356] "Generic (PLEG): container finished" podID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerID="34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb" exitCode=0
Sep 09 20:36:19.525338 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.525308    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"
Sep 09 20:36:19.525836 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.525343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" event={"ID":"32b28779-e0dd-4d7a-be0f-8216b7f86752","Type":"ContainerDied","Data":"34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb"}
Sep 09 20:36:19.525836 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.525398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg" event={"ID":"32b28779-e0dd-4d7a-be0f-8216b7f86752","Type":"ContainerDied","Data":"d9b0b584cc1758b8a183958d06d67e2328f8e62a413b8f8712797fa0b128129c"}
Sep 09 20:36:19.525836 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.525429    2562 scope.go:117] "RemoveContainer" containerID="e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6"
Sep 09 20:36:19.525980 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.525886    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:36:19.533730 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.533713    2562 scope.go:117] "RemoveContainer" containerID="34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb"
Sep 09 20:36:19.540753 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.540733    2562 scope.go:117] "RemoveContainer" containerID="e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6"
Sep 09 20:36:19.541028 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:36:19.540997    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6\": container with ID starting with e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6 not found: ID does not exist" containerID="e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6"
Sep 09 20:36:19.541084 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.541036    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6"} err="failed to get container status \"e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6\": rpc error: code = NotFound desc = could not find container \"e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6\": container with ID starting with e52d25f95f9e6e04f9abdb40b6ea09c4f51b8d564bb7cf2d1facf39fa1b443a6 not found: ID does not exist"
Sep 09 20:36:19.541084 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.541052    2562 scope.go:117] "RemoveContainer" containerID="34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb"
Sep 09 20:36:19.541309 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:36:19.541292    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb\": container with ID starting with 34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb not found: ID does not exist" containerID="34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb"
Sep 09 20:36:19.541380 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.541315    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb"} err="failed to get container status \"34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb\": rpc error: code = NotFound desc = could not find container \"34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb\": container with ID starting with 34714a25f776f08e5f468ff831d3fd2962fab8277bceab22bfbd5e36674414eb not found: ID does not exist"
Sep 09 20:36:19.546878 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.546854    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"]
Sep 09 20:36:19.550268 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:19.550247    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53cb1-predictor-77bd468459-7gnsg"]
Sep 09 20:36:20.466118 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:20.466082    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" path="/var/lib/kubelet/pods/32b28779-e0dd-4d7a-be0f-8216b7f86752/volumes"
Sep 09 20:36:23.335342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:23.335306    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:24.489815 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:24.489790    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:36:24.492392 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:24.492367    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:36:24.531490 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:24.531466    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:36:24.532072 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:24.532048    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:36:26.291816 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.291782    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"]
Sep 09 20:36:26.292207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.292127    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container"
Sep 09 20:36:26.292207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.292140    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container"
Sep 09 20:36:26.292207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.292158    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kube-rbac-proxy"
Sep 09 20:36:26.292207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.292163    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kube-rbac-proxy"
Sep 09 20:36:26.292344 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.292220    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kserve-container"
Sep 09 20:36:26.292344 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.292229    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b28779-e0dd-4d7a-be0f-8216b7f86752" containerName="kube-rbac-proxy"
Sep 09 20:36:26.294935 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.294917    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.297123 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.297106    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b3d7b-serving-cert\""
Sep 09 20:36:26.297218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.297106    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b3d7b-kube-rbac-proxy-sar-config\""
Sep 09 20:36:26.303317 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.303299    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"]
Sep 09 20:36:26.382281 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.382253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26fea029-969d-4311-a62d-b44401340980-openshift-service-ca-bundle\") pod \"ensemble-graph-b3d7b-6c989bffcf-kgvbk\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") " pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.382395 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.382293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26fea029-969d-4311-a62d-b44401340980-proxy-tls\") pod \"ensemble-graph-b3d7b-6c989bffcf-kgvbk\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") " pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.483305 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.483280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26fea029-969d-4311-a62d-b44401340980-openshift-service-ca-bundle\") pod \"ensemble-graph-b3d7b-6c989bffcf-kgvbk\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") " pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.483418 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.483317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26fea029-969d-4311-a62d-b44401340980-proxy-tls\") pod \"ensemble-graph-b3d7b-6c989bffcf-kgvbk\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") " pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.483909 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.483887    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26fea029-969d-4311-a62d-b44401340980-openshift-service-ca-bundle\") pod \"ensemble-graph-b3d7b-6c989bffcf-kgvbk\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") " pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.485559 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.485543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26fea029-969d-4311-a62d-b44401340980-proxy-tls\") pod \"ensemble-graph-b3d7b-6c989bffcf-kgvbk\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") " pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.605668 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.605608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:26.874796 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:26.874769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"]
Sep 09 20:36:26.876965 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:36:26.876943    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fea029_969d_4311_a62d_b44401340980.slice/crio-01b1b99afa861fb0140269f0620e53c624701bcfca7a71afd77119607ce82ac2 WatchSource:0}: Error finding container 01b1b99afa861fb0140269f0620e53c624701bcfca7a71afd77119607ce82ac2: Status 404 returned error can't find the container with id 01b1b99afa861fb0140269f0620e53c624701bcfca7a71afd77119607ce82ac2
Sep 09 20:36:27.561200 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:27.561168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" event={"ID":"26fea029-969d-4311-a62d-b44401340980","Type":"ContainerStarted","Data":"c58d64ad48e593aea59ec1363884017385f2b2ce6cd49c566d251054d2e2a3cc"}
Sep 09 20:36:27.561563 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:27.561219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" event={"ID":"26fea029-969d-4311-a62d-b44401340980","Type":"ContainerStarted","Data":"01b1b99afa861fb0140269f0620e53c624701bcfca7a71afd77119607ce82ac2"}
Sep 09 20:36:27.561563 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:27.561294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:27.576542 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:27.576498    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podStartSLOduration=1.576484297 podStartE2EDuration="1.576484297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:27.574974178 +0000 UTC m=+1503.548744242" watchObservedRunningTime="2026-09-09 20:36:27.576484297 +0000 UTC m=+1503.550254392"
Sep 09 20:36:28.335041 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:28.334986    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:28.335201 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:28.335123    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:36:33.334603 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:33.334568    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:33.571034 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:33.570992    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:34.532912 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:34.532876    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:36:36.365589 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.365499    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"]
Sep 09 20:36:36.365962 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.365739    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b" containerID="cri-o://c58d64ad48e593aea59ec1363884017385f2b2ce6cd49c566d251054d2e2a3cc" gracePeriod=30
Sep 09 20:36:36.452198 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.452168    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"]
Sep 09 20:36:36.452470 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.452426    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container" containerID="cri-o://f554ebc0028f6310de2f0c27c23a56f4141ada1fefb4c07307025609da220153" gracePeriod=30
Sep 09 20:36:36.452553 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.452457    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kube-rbac-proxy" containerID="cri-o://cbd1f27f8595782fbb593254653febb9f206ce3b81d98043b9af9bf472b77951" gracePeriod=30
Sep 09 20:36:36.574979 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.574948    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"]
Sep 09 20:36:36.577993 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.577977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.580295 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.580273    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-81279-predictor-serving-cert\""
Sep 09 20:36:36.580295 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.580287    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-81279-kube-rbac-proxy-sar-config\""
Sep 09 20:36:36.586972 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.586951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"]
Sep 09 20:36:36.602539 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.602428    2562 generic.go:356] "Generic (PLEG): container finished" podID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerID="cbd1f27f8595782fbb593254653febb9f206ce3b81d98043b9af9bf472b77951" exitCode=2
Sep 09 20:36:36.602539 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.602452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" event={"ID":"142fe19b-06bd-4efa-a0a3-bb06d998517d","Type":"ContainerDied","Data":"cbd1f27f8595782fbb593254653febb9f206ce3b81d98043b9af9bf472b77951"}
Sep 09 20:36:36.661066 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.661040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.661185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.661111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7b5t\" (UniqueName: \"kubernetes.io/projected/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-api-access-p7b5t\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.661185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.661133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.761884 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.761859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.761987 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.761897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.761987 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.761949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7b5t\" (UniqueName: \"kubernetes.io/projected/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-api-access-p7b5t\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.762099 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:36:36.762078    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-81279-predictor-serving-cert: secret "success-200-isvc-81279-predictor-serving-cert" not found
Sep 09 20:36:36.762172 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:36:36.762159    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls podName:fd0dffb6-8f17-488c-93cf-5ec5c832153a nodeName:}" failed. No retries permitted until 2026-09-09 20:36:37.262134442 +0000 UTC m=+1513.235904486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls") pod "success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" (UID: "fd0dffb6-8f17-488c-93cf-5ec5c832153a") : secret "success-200-isvc-81279-predictor-serving-cert" not found
Sep 09 20:36:36.762569 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.762551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:36.772294 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:36.772274    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7b5t\" (UniqueName: \"kubernetes.io/projected/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-api-access-p7b5t\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:37.266187 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:37.266151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:37.268469 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:37.268446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls\") pod \"success-200-isvc-81279-predictor-bb8f78f5c-qlsqq\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") " pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:37.488650 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:37.488613    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:37.758013 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:37.757972    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"]
Sep 09 20:36:37.760563 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:36:37.760533    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0dffb6_8f17_488c_93cf_5ec5c832153a.slice/crio-bc9fa3dd83462cdc725d17b2249006607d821b6b8b18e97338fcd967fab9b65c WatchSource:0}: Error finding container bc9fa3dd83462cdc725d17b2249006607d821b6b8b18e97338fcd967fab9b65c: Status 404 returned error can't find the container with id bc9fa3dd83462cdc725d17b2249006607d821b6b8b18e97338fcd967fab9b65c
Sep 09 20:36:38.335277 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:38.335241    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:38.568620 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:38.568585    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:38.617538 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:38.617453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" event={"ID":"fd0dffb6-8f17-488c-93cf-5ec5c832153a","Type":"ContainerStarted","Data":"7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46"}
Sep 09 20:36:38.617538 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:38.617518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" event={"ID":"fd0dffb6-8f17-488c-93cf-5ec5c832153a","Type":"ContainerStarted","Data":"cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef"}
Sep 09 20:36:38.617680 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:38.617552    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:38.617680 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:38.617580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" event={"ID":"fd0dffb6-8f17-488c-93cf-5ec5c832153a","Type":"ContainerStarted","Data":"bc9fa3dd83462cdc725d17b2249006607d821b6b8b18e97338fcd967fab9b65c"}
Sep 09 20:36:38.634965 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:38.634918    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podStartSLOduration=2.634903882 podStartE2EDuration="2.634903882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:38.633507055 +0000 UTC m=+1514.607277117" watchObservedRunningTime="2026-09-09 20:36:38.634903882 +0000 UTC m=+1514.608673945"
Sep 09 20:36:39.624815 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.624777    2562 generic.go:356] "Generic (PLEG): container finished" podID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerID="f554ebc0028f6310de2f0c27c23a56f4141ada1fefb4c07307025609da220153" exitCode=0
Sep 09 20:36:39.625244 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.624842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" event={"ID":"142fe19b-06bd-4efa-a0a3-bb06d998517d","Type":"ContainerDied","Data":"f554ebc0028f6310de2f0c27c23a56f4141ada1fefb4c07307025609da220153"}
Sep 09 20:36:39.625244 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.625212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:39.626452 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.626428    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:39.703174 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.703154    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:36:39.786073 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.786011    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2stmx\" (UniqueName: \"kubernetes.io/projected/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-api-access-2stmx\") pod \"142fe19b-06bd-4efa-a0a3-bb06d998517d\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") "
Sep 09 20:36:39.786073 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.786059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls\") pod \"142fe19b-06bd-4efa-a0a3-bb06d998517d\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") "
Sep 09 20:36:39.786270 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.786116    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-rbac-proxy-sar-config\") pod \"142fe19b-06bd-4efa-a0a3-bb06d998517d\" (UID: \"142fe19b-06bd-4efa-a0a3-bb06d998517d\") "
Sep 09 20:36:39.786531 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.786506    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "142fe19b-06bd-4efa-a0a3-bb06d998517d" (UID: "142fe19b-06bd-4efa-a0a3-bb06d998517d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:39.788064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.788040    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "142fe19b-06bd-4efa-a0a3-bb06d998517d" (UID: "142fe19b-06bd-4efa-a0a3-bb06d998517d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:39.788153 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.788058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-api-access-2stmx" (OuterVolumeSpecName: "kube-api-access-2stmx") pod "142fe19b-06bd-4efa-a0a3-bb06d998517d" (UID: "142fe19b-06bd-4efa-a0a3-bb06d998517d"). InnerVolumeSpecName "kube-api-access-2stmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:39.886639 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.886613    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stmx\" (UniqueName: \"kubernetes.io/projected/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-api-access-2stmx\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:39.886639 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.886636    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/142fe19b-06bd-4efa-a0a3-bb06d998517d-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:39.886639 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:39.886647    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/142fe19b-06bd-4efa-a0a3-bb06d998517d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:40.631210 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:40.631183    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"
Sep 09 20:36:40.631577 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:40.631219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6" event={"ID":"142fe19b-06bd-4efa-a0a3-bb06d998517d","Type":"ContainerDied","Data":"b11085877e243452aca94f7bed19f208dddcef7127898638c5876290241ef6c6"}
Sep 09 20:36:40.631577 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:40.631282    2562 scope.go:117] "RemoveContainer" containerID="cbd1f27f8595782fbb593254653febb9f206ce3b81d98043b9af9bf472b77951"
Sep 09 20:36:40.631871 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:40.631845    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:40.639810 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:40.639790    2562 scope.go:117] "RemoveContainer" containerID="f554ebc0028f6310de2f0c27c23a56f4141ada1fefb4c07307025609da220153"
Sep 09 20:36:40.648287 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:40.648267    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"]
Sep 09 20:36:40.651917 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:40.651898    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3d7b-predictor-5b7c8bf675-xwlk6"]
Sep 09 20:36:42.465497 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:42.465467    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" path="/var/lib/kubelet/pods/142fe19b-06bd-4efa-a0a3-bb06d998517d/volumes"
Sep 09 20:36:43.335600 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:43.335565    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:43.568710 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:43.568676    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:44.532244 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:44.532210    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:36:45.636778 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:45.636740    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:36:45.637229 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:45.637204    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:46.226887 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.226858    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:36:46.337634 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.337569    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c756ac-a262-48aa-9467-34510e4b9602-openshift-service-ca-bundle\") pod \"56c756ac-a262-48aa-9467-34510e4b9602\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") "
Sep 09 20:36:46.337786 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.337659    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c756ac-a262-48aa-9467-34510e4b9602-proxy-tls\") pod \"56c756ac-a262-48aa-9467-34510e4b9602\" (UID: \"56c756ac-a262-48aa-9467-34510e4b9602\") "
Sep 09 20:36:46.337934 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.337912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c756ac-a262-48aa-9467-34510e4b9602-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "56c756ac-a262-48aa-9467-34510e4b9602" (UID: "56c756ac-a262-48aa-9467-34510e4b9602"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:46.339738 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.339716    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c756ac-a262-48aa-9467-34510e4b9602-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56c756ac-a262-48aa-9467-34510e4b9602" (UID: "56c756ac-a262-48aa-9467-34510e4b9602"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:46.439187 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.439151    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c756ac-a262-48aa-9467-34510e4b9602-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:46.439187 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.439185    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c756ac-a262-48aa-9467-34510e4b9602-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:36:46.660688 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.660653    2562 generic.go:356] "Generic (PLEG): container finished" podID="56c756ac-a262-48aa-9467-34510e4b9602" containerID="b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059" exitCode=0
Sep 09 20:36:46.661100 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.660700    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"
Sep 09 20:36:46.661100 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.660724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" event={"ID":"56c756ac-a262-48aa-9467-34510e4b9602","Type":"ContainerDied","Data":"b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059"}
Sep 09 20:36:46.661100 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.660774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb" event={"ID":"56c756ac-a262-48aa-9467-34510e4b9602","Type":"ContainerDied","Data":"9dea81a423b7154ccef908b7d61d2e06e11f55da2182e43b67414d830159e1dd"}
Sep 09 20:36:46.661100 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.660803    2562 scope.go:117] "RemoveContainer" containerID="b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059"
Sep 09 20:36:46.668862 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.668839    2562 scope.go:117] "RemoveContainer" containerID="b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059"
Sep 09 20:36:46.669134 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:36:46.669113    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059\": container with ID starting with b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059 not found: ID does not exist" containerID="b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059"
Sep 09 20:36:46.669204 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.669140    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059"} err="failed to get container status \"b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059\": rpc error: code = NotFound desc = could not find container \"b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059\": container with ID starting with b2a28997c3d79962eff2ba604588166b3263dafb64652f94782ca8835d727059 not found: ID does not exist"
Sep 09 20:36:46.676561 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.676537    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"]
Sep 09 20:36:46.679850 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:46.679827    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53cb1-d889798ff-6bkzb"]
Sep 09 20:36:48.465725 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:48.465689    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c756ac-a262-48aa-9467-34510e4b9602" path="/var/lib/kubelet/pods/56c756ac-a262-48aa-9467-34510e4b9602/volumes"
Sep 09 20:36:48.569259 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:48.569226    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:48.569417 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:48.569324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:36:53.569143 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:53.569104    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:54.533069 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:54.533033    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 20:36:55.637797 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:55.637755    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:58.569092 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:36:58.569052    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:03.569492 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:03.569450    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:04.532906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:04.532878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:37:05.637227 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:05.637187    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:37:06.745738 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:06.745705    2562 generic.go:356] "Generic (PLEG): container finished" podID="26fea029-969d-4311-a62d-b44401340980" containerID="c58d64ad48e593aea59ec1363884017385f2b2ce6cd49c566d251054d2e2a3cc" exitCode=0
Sep 09 20:37:06.746168 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:06.745782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" event={"ID":"26fea029-969d-4311-a62d-b44401340980","Type":"ContainerDied","Data":"c58d64ad48e593aea59ec1363884017385f2b2ce6cd49c566d251054d2e2a3cc"}
Sep 09 20:37:07.016519 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.016495    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:37:07.104906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.104868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26fea029-969d-4311-a62d-b44401340980-openshift-service-ca-bundle\") pod \"26fea029-969d-4311-a62d-b44401340980\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") "
Sep 09 20:37:07.105082 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.104940    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26fea029-969d-4311-a62d-b44401340980-proxy-tls\") pod \"26fea029-969d-4311-a62d-b44401340980\" (UID: \"26fea029-969d-4311-a62d-b44401340980\") "
Sep 09 20:37:07.105319 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.105296    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fea029-969d-4311-a62d-b44401340980-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "26fea029-969d-4311-a62d-b44401340980" (UID: "26fea029-969d-4311-a62d-b44401340980"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:07.107012 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.106974    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fea029-969d-4311-a62d-b44401340980-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26fea029-969d-4311-a62d-b44401340980" (UID: "26fea029-969d-4311-a62d-b44401340980"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:07.206070 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.206028    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26fea029-969d-4311-a62d-b44401340980-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:37:07.206070 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.206063    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26fea029-969d-4311-a62d-b44401340980-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:37:07.752558 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.752527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk" event={"ID":"26fea029-969d-4311-a62d-b44401340980","Type":"ContainerDied","Data":"01b1b99afa861fb0140269f0620e53c624701bcfca7a71afd77119607ce82ac2"}
Sep 09 20:37:07.752971 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.752590    2562 scope.go:117] "RemoveContainer" containerID="c58d64ad48e593aea59ec1363884017385f2b2ce6cd49c566d251054d2e2a3cc"
Sep 09 20:37:07.752971 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.752539    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"
Sep 09 20:37:07.774776 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.774749    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"]
Sep 09 20:37:07.779390 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:07.779364    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b3d7b-6c989bffcf-kgvbk"]
Sep 09 20:37:08.471256 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:08.471221    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fea029-969d-4311-a62d-b44401340980" path="/var/lib/kubelet/pods/26fea029-969d-4311-a62d-b44401340980/volumes"
Sep 09 20:37:15.637265 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:15.637227    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:37:16.278523 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278493    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"]
Sep 09 20:37:16.278853 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278839    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container"
Sep 09 20:37:16.278910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278854    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container"
Sep 09 20:37:16.278910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278867    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1"
Sep 09 20:37:16.278910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278873    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1"
Sep 09 20:37:16.278910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278884    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kube-rbac-proxy"
Sep 09 20:37:16.278910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278891    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kube-rbac-proxy"
Sep 09 20:37:16.278910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278901    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b"
Sep 09 20:37:16.278910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278907    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b"
Sep 09 20:37:16.279149 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278960    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kube-rbac-proxy"
Sep 09 20:37:16.279149 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278967    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="142fe19b-06bd-4efa-a0a3-bb06d998517d" containerName="kserve-container"
Sep 09 20:37:16.279149 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278974    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fea029-969d-4311-a62d-b44401340980" containerName="ensemble-graph-b3d7b"
Sep 09 20:37:16.279149 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.278980    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c756ac-a262-48aa-9467-34510e4b9602" containerName="sequence-graph-53cb1"
Sep 09 20:37:16.281509 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.281494    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:16.284130 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.284108    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-90077-serving-cert\""
Sep 09 20:37:16.284251 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.284115    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-90077-kube-rbac-proxy-sar-config\""
Sep 09 20:37:16.289706 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.289684    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"]
Sep 09 20:37:16.379604 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.379578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls\") pod \"sequence-graph-90077-7b4844c498-zld4d\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") " pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:16.379726 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.379612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78b9e251-263d-4223-a108-cb681039c938-openshift-service-ca-bundle\") pod \"sequence-graph-90077-7b4844c498-zld4d\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") " pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:16.480159 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.480132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls\") pod \"sequence-graph-90077-7b4844c498-zld4d\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") " pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:16.480280 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.480165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78b9e251-263d-4223-a108-cb681039c938-openshift-service-ca-bundle\") pod \"sequence-graph-90077-7b4844c498-zld4d\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") " pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:16.480280 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:37:16.480265    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-90077-serving-cert: secret "sequence-graph-90077-serving-cert" not found
Sep 09 20:37:16.480351 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:37:16.480329    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls podName:78b9e251-263d-4223-a108-cb681039c938 nodeName:}" failed. No retries permitted until 2026-09-09 20:37:16.980314237 +0000 UTC m=+1552.954084278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls") pod "sequence-graph-90077-7b4844c498-zld4d" (UID: "78b9e251-263d-4223-a108-cb681039c938") : secret "sequence-graph-90077-serving-cert" not found
Sep 09 20:37:16.480747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.480731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78b9e251-263d-4223-a108-cb681039c938-openshift-service-ca-bundle\") pod \"sequence-graph-90077-7b4844c498-zld4d\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") " pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:16.982483 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.982448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls\") pod \"sequence-graph-90077-7b4844c498-zld4d\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") " pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:16.984747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:16.984723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls\") pod \"sequence-graph-90077-7b4844c498-zld4d\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") " pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:17.192411 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:17.192381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:17.460395 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:17.460308    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"]
Sep 09 20:37:17.462578 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:37:17.462550    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b9e251_263d_4223_a108_cb681039c938.slice/crio-b4f22296f6f7653f7dce2421beb3019545811804b9451e5d584503e595c69433 WatchSource:0}: Error finding container b4f22296f6f7653f7dce2421beb3019545811804b9451e5d584503e595c69433: Status 404 returned error can't find the container with id b4f22296f6f7653f7dce2421beb3019545811804b9451e5d584503e595c69433
Sep 09 20:37:17.798281 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:17.798200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" event={"ID":"78b9e251-263d-4223-a108-cb681039c938","Type":"ContainerStarted","Data":"1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c"}
Sep 09 20:37:17.798281 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:17.798249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" event={"ID":"78b9e251-263d-4223-a108-cb681039c938","Type":"ContainerStarted","Data":"b4f22296f6f7653f7dce2421beb3019545811804b9451e5d584503e595c69433"}
Sep 09 20:37:17.798452 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:17.798282    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:17.815562 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:17.815513    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podStartSLOduration=1.815497777 podStartE2EDuration="1.815497777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:17.814243117 +0000 UTC m=+1553.788013180" watchObservedRunningTime="2026-09-09 20:37:17.815497777 +0000 UTC m=+1553.789267840"
Sep 09 20:37:23.808175 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:23.808147    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:25.638108 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:25.638081    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:37:26.352496 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.352459    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"]
Sep 09 20:37:26.352774 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.352730    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077" containerID="cri-o://1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c" gracePeriod=30
Sep 09 20:37:26.445446 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.445410    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"]
Sep 09 20:37:26.445661 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.445620    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container" containerID="cri-o://24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa" gracePeriod=30
Sep 09 20:37:26.445756 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.445652    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kube-rbac-proxy" containerID="cri-o://1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59" gracePeriod=30
Sep 09 20:37:26.539350 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.539315    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"]
Sep 09 20:37:26.542721 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.542706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.545390 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.545366    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-28e88-kube-rbac-proxy-sar-config\""
Sep 09 20:37:26.545496 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.545406    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-28e88-predictor-serving-cert\""
Sep 09 20:37:26.552940 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.552919    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"]
Sep 09 20:37:26.661241 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.661211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p5v\" (UniqueName: \"kubernetes.io/projected/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-api-access-d8p5v\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.661587 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.661264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb665308-c203-4bd9-aedc-9d9078d099ef-proxy-tls\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.661587 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.661330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.762307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.762275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.762456 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.762320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p5v\" (UniqueName: \"kubernetes.io/projected/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-api-access-d8p5v\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.762456 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.762361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb665308-c203-4bd9-aedc-9d9078d099ef-proxy-tls\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.762969 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.762949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.764742 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.764724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb665308-c203-4bd9-aedc-9d9078d099ef-proxy-tls\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.770964 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.770941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p5v\" (UniqueName: \"kubernetes.io/projected/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-api-access-d8p5v\") pod \"success-200-isvc-28e88-predictor-779f848669-jfwgg\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") " pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:26.837522 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.837486    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerID="1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59" exitCode=2
Sep 09 20:37:26.837647 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.837562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" event={"ID":"d1d2a4bd-d2f8-489c-a578-59501bda5835","Type":"ContainerDied","Data":"1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59"}
Sep 09 20:37:26.852879 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:26.852856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:27.121182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.121107    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"]
Sep 09 20:37:27.123325 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:37:27.123285    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb665308_c203_4bd9_aedc_9d9078d099ef.slice/crio-6d25e201c648f6c5ca4e5becb78e2fd8f03f3470d6a4ab6c7c7ccffc6d12ec22 WatchSource:0}: Error finding container 6d25e201c648f6c5ca4e5becb78e2fd8f03f3470d6a4ab6c7c7ccffc6d12ec22: Status 404 returned error can't find the container with id 6d25e201c648f6c5ca4e5becb78e2fd8f03f3470d6a4ab6c7c7ccffc6d12ec22
Sep 09 20:37:27.843859 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.843826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" event={"ID":"bb665308-c203-4bd9-aedc-9d9078d099ef","Type":"ContainerStarted","Data":"1f8fe6cd1ebb6c0c9e078f33326a536831a1c41a5514e9974160d125b32fec8d"}
Sep 09 20:37:27.844299 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.843880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" event={"ID":"bb665308-c203-4bd9-aedc-9d9078d099ef","Type":"ContainerStarted","Data":"4add40fd9d92712acda5dc68daa549ae263e48e860e4d17fc5e548ad16451a2b"}
Sep 09 20:37:27.844299 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.843907    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:27.844299 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.843925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" event={"ID":"bb665308-c203-4bd9-aedc-9d9078d099ef","Type":"ContainerStarted","Data":"6d25e201c648f6c5ca4e5becb78e2fd8f03f3470d6a4ab6c7c7ccffc6d12ec22"}
Sep 09 20:37:27.844299 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.843975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:27.845247 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.845220    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 20:37:27.860465 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:27.860422    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podStartSLOduration=1.860411815 podStartE2EDuration="1.860411815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:27.858965211 +0000 UTC m=+1563.832735274" watchObservedRunningTime="2026-09-09 20:37:27.860411815 +0000 UTC m=+1563.834181878"
Sep 09 20:37:28.807499 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:28.807461    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:28.847784 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:28.847747    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 20:37:29.287859 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.287832    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:37:29.385305 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.385278    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls\") pod \"d1d2a4bd-d2f8-489c-a578-59501bda5835\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") "
Sep 09 20:37:29.385444 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.385317    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-rbac-proxy-sar-config\") pod \"d1d2a4bd-d2f8-489c-a578-59501bda5835\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") "
Sep 09 20:37:29.385444 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.385373    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8jl4\" (UniqueName: \"kubernetes.io/projected/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-api-access-s8jl4\") pod \"d1d2a4bd-d2f8-489c-a578-59501bda5835\" (UID: \"d1d2a4bd-d2f8-489c-a578-59501bda5835\") "
Sep 09 20:37:29.385717 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.385694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1d2a4bd-d2f8-489c-a578-59501bda5835" (UID: "d1d2a4bd-d2f8-489c-a578-59501bda5835"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:29.387414 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.387392    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-api-access-s8jl4" (OuterVolumeSpecName: "kube-api-access-s8jl4") pod "d1d2a4bd-d2f8-489c-a578-59501bda5835" (UID: "d1d2a4bd-d2f8-489c-a578-59501bda5835"). InnerVolumeSpecName "kube-api-access-s8jl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:29.387511 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.387411    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1d2a4bd-d2f8-489c-a578-59501bda5835" (UID: "d1d2a4bd-d2f8-489c-a578-59501bda5835"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:29.486932 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.486903    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8jl4\" (UniqueName: \"kubernetes.io/projected/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-api-access-s8jl4\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:37:29.486932 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.486927    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d2a4bd-d2f8-489c-a578-59501bda5835-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:37:29.486932 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.486937    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1d2a4bd-d2f8-489c-a578-59501bda5835-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:37:29.853458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.853311    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerID="24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa" exitCode=0
Sep 09 20:37:29.853458 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.853430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" event={"ID":"d1d2a4bd-d2f8-489c-a578-59501bda5835","Type":"ContainerDied","Data":"24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa"}
Sep 09 20:37:29.853863 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.853462    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"
Sep 09 20:37:29.853863 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.853496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk" event={"ID":"d1d2a4bd-d2f8-489c-a578-59501bda5835","Type":"ContainerDied","Data":"cf9f6ec40567ca8df7af908f48bf75d5c17070349beee0052837ab978bf9a9e5"}
Sep 09 20:37:29.853863 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.853539    2562 scope.go:117] "RemoveContainer" containerID="1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59"
Sep 09 20:37:29.861648 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.861627    2562 scope.go:117] "RemoveContainer" containerID="24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa"
Sep 09 20:37:29.869206 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.869189    2562 scope.go:117] "RemoveContainer" containerID="1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59"
Sep 09 20:37:29.869482 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:37:29.869464    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59\": container with ID starting with 1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59 not found: ID does not exist" containerID="1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59"
Sep 09 20:37:29.869534 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.869490    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59"} err="failed to get container status \"1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59\": rpc error: code = NotFound desc = could not find container \"1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59\": container with ID starting with 1dfbf63275ef3155ddb767b2d781d695a34e6b80723e883bfbaf1e9502318b59 not found: ID does not exist"
Sep 09 20:37:29.869534 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.869506    2562 scope.go:117] "RemoveContainer" containerID="24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa"
Sep 09 20:37:29.869747 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:37:29.869726    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa\": container with ID starting with 24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa not found: ID does not exist" containerID="24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa"
Sep 09 20:37:29.869805 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.869757    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa"} err="failed to get container status \"24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa\": rpc error: code = NotFound desc = could not find container \"24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa\": container with ID starting with 24bb112b339bb12f478d5bac38cf1db9dd025828ae5fe0045049b9af6dab18fa not found: ID does not exist"
Sep 09 20:37:29.873988 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.873963    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"]
Sep 09 20:37:29.878882 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:29.878860    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90077-predictor-58445bbccc-n7mjk"]
Sep 09 20:37:30.465842 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:30.465809    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" path="/var/lib/kubelet/pods/d1d2a4bd-d2f8-489c-a578-59501bda5835/volumes"
Sep 09 20:37:33.806995 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:33.806948    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:33.852449 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:33.852425    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:37:33.852882 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:33.852856    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 20:37:36.598978 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.598943    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"]
Sep 09 20:37:36.599338 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.599276    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kube-rbac-proxy"
Sep 09 20:37:36.599338 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.599288    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kube-rbac-proxy"
Sep 09 20:37:36.599338 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.599305    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container"
Sep 09 20:37:36.599338 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.599311    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container"
Sep 09 20:37:36.599461 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.599365    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kube-rbac-proxy"
Sep 09 20:37:36.599461 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.599372    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d2a4bd-d2f8-489c-a578-59501bda5835" containerName="kserve-container"
Sep 09 20:37:36.603854 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.603838    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:36.606152 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.606127    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-81279-serving-cert\""
Sep 09 20:37:36.606152 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.606142    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-81279-kube-rbac-proxy-sar-config\""
Sep 09 20:37:36.610239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.610214    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"]
Sep 09 20:37:36.746130 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.746098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls\") pod \"ensemble-graph-81279-7ff94b5fb-vhq95\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") " pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:36.746273 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.746141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4461df58-2188-41a6-90a5-0c6b8e77aee2-openshift-service-ca-bundle\") pod \"ensemble-graph-81279-7ff94b5fb-vhq95\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") " pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:36.847096 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.847071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4461df58-2188-41a6-90a5-0c6b8e77aee2-openshift-service-ca-bundle\") pod \"ensemble-graph-81279-7ff94b5fb-vhq95\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") " pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:36.847202 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.847140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls\") pod \"ensemble-graph-81279-7ff94b5fb-vhq95\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") " pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:36.847263 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:37:36.847227    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-81279-serving-cert: secret "ensemble-graph-81279-serving-cert" not found
Sep 09 20:37:36.847299 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:37:36.847286    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls podName:4461df58-2188-41a6-90a5-0c6b8e77aee2 nodeName:}" failed. No retries permitted until 2026-09-09 20:37:37.347271436 +0000 UTC m=+1573.321041482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls") pod "ensemble-graph-81279-7ff94b5fb-vhq95" (UID: "4461df58-2188-41a6-90a5-0c6b8e77aee2") : secret "ensemble-graph-81279-serving-cert" not found
Sep 09 20:37:36.847706 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:36.847689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4461df58-2188-41a6-90a5-0c6b8e77aee2-openshift-service-ca-bundle\") pod \"ensemble-graph-81279-7ff94b5fb-vhq95\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") " pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:37.350562 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:37.350529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls\") pod \"ensemble-graph-81279-7ff94b5fb-vhq95\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") " pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:37.352883 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:37.352858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls\") pod \"ensemble-graph-81279-7ff94b5fb-vhq95\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") " pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:37.515195 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:37.515162    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:37.785307 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:37.785282    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"]
Sep 09 20:37:37.789888 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:37:37.789548    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4461df58_2188_41a6_90a5_0c6b8e77aee2.slice/crio-31dad35ec538a1adca1f3a195ad043ac4d3c20a041d72d851471a58c2bea83d5 WatchSource:0}: Error finding container 31dad35ec538a1adca1f3a195ad043ac4d3c20a041d72d851471a58c2bea83d5: Status 404 returned error can't find the container with id 31dad35ec538a1adca1f3a195ad043ac4d3c20a041d72d851471a58c2bea83d5
Sep 09 20:37:37.890714 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:37.890678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" event={"ID":"4461df58-2188-41a6-90a5-0c6b8e77aee2","Type":"ContainerStarted","Data":"263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084"}
Sep 09 20:37:37.890836 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:37.890750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" event={"ID":"4461df58-2188-41a6-90a5-0c6b8e77aee2","Type":"ContainerStarted","Data":"31dad35ec538a1adca1f3a195ad043ac4d3c20a041d72d851471a58c2bea83d5"}
Sep 09 20:37:37.890836 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:37.890781    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:38.807105 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:38.807063    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:38.807536 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:38.807195    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:38.823592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:38.823541    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podStartSLOduration=2.823526352 podStartE2EDuration="2.823526352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:37.905803127 +0000 UTC m=+1573.879573189" watchObservedRunningTime="2026-09-09 20:37:38.823526352 +0000 UTC m=+1574.797296414"
Sep 09 20:37:43.806747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:43.806702    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:43.853137 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:43.853095    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 20:37:43.900346 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:43.900318    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:37:48.807035 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:48.806962    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:53.806859 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:53.806813    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:53.853426 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:53.853386    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 20:37:56.534398 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.534373    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:56.617294 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.617224    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls\") pod \"78b9e251-263d-4223-a108-cb681039c938\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") "
Sep 09 20:37:56.617294 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.617277    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78b9e251-263d-4223-a108-cb681039c938-openshift-service-ca-bundle\") pod \"78b9e251-263d-4223-a108-cb681039c938\" (UID: \"78b9e251-263d-4223-a108-cb681039c938\") "
Sep 09 20:37:56.617655 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.617628    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b9e251-263d-4223-a108-cb681039c938-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "78b9e251-263d-4223-a108-cb681039c938" (UID: "78b9e251-263d-4223-a108-cb681039c938"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:56.619202 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.619178    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78b9e251-263d-4223-a108-cb681039c938" (UID: "78b9e251-263d-4223-a108-cb681039c938"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:56.718730 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.718697    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78b9e251-263d-4223-a108-cb681039c938-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:37:56.718730 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.718726    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b9e251-263d-4223-a108-cb681039c938-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:37:56.976013 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.975975    2562 generic.go:356] "Generic (PLEG): container finished" podID="78b9e251-263d-4223-a108-cb681039c938" containerID="1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c" exitCode=0
Sep 09 20:37:56.976207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.976031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" event={"ID":"78b9e251-263d-4223-a108-cb681039c938","Type":"ContainerDied","Data":"1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c"}
Sep 09 20:37:56.976207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.976038    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"
Sep 09 20:37:56.976207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.976099    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d" event={"ID":"78b9e251-263d-4223-a108-cb681039c938","Type":"ContainerDied","Data":"b4f22296f6f7653f7dce2421beb3019545811804b9451e5d584503e595c69433"}
Sep 09 20:37:56.976207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.976120    2562 scope.go:117] "RemoveContainer" containerID="1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c"
Sep 09 20:37:56.985042 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.985023    2562 scope.go:117] "RemoveContainer" containerID="1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c"
Sep 09 20:37:56.985315 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:37:56.985295    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c\": container with ID starting with 1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c not found: ID does not exist" containerID="1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c"
Sep 09 20:37:56.985364 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.985323    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c"} err="failed to get container status \"1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c\": rpc error: code = NotFound desc = could not find container \"1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c\": container with ID starting with 1909dab478fad003a3c79835f2a1d5fbe55913fe971c3eb49035f1ef2f3b379c not found: ID does not exist"
Sep 09 20:37:56.996941 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:56.996916    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"]
Sep 09 20:37:57.002355 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:57.002332    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-90077-7b4844c498-zld4d"]
Sep 09 20:37:58.465731 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:37:58.465696    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b9e251-263d-4223-a108-cb681039c938" path="/var/lib/kubelet/pods/78b9e251-263d-4223-a108-cb681039c938/volumes"
Sep 09 20:38:03.852974 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:03.852933    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 20:38:13.853587 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:13.853555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:38:26.607543 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.607512    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"]
Sep 09 20:38:26.607910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.607827    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077"
Sep 09 20:38:26.607910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.607838    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077"
Sep 09 20:38:26.607910 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.607906    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b9e251-263d-4223-a108-cb681039c938" containerName="sequence-graph-90077"
Sep 09 20:38:26.610583 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.610566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:26.613283 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.613256    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-28e88-kube-rbac-proxy-sar-config\""
Sep 09 20:38:26.613408 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.613308    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-28e88-serving-cert\""
Sep 09 20:38:26.620924 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.620903    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"]
Sep 09 20:38:26.763138 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.763112    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-openshift-service-ca-bundle\") pod \"sequence-graph-28e88-f549854fc-7qtn8\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") " pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:26.763273 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.763141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-proxy-tls\") pod \"sequence-graph-28e88-f549854fc-7qtn8\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") " pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:26.864041 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.863950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-openshift-service-ca-bundle\") pod \"sequence-graph-28e88-f549854fc-7qtn8\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") " pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:26.864041 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.863981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-proxy-tls\") pod \"sequence-graph-28e88-f549854fc-7qtn8\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") " pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:26.864549 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.864530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-openshift-service-ca-bundle\") pod \"sequence-graph-28e88-f549854fc-7qtn8\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") " pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:26.866341 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.866321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-proxy-tls\") pod \"sequence-graph-28e88-f549854fc-7qtn8\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") " pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:26.921147 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:26.921107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:27.188793 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:27.188768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"]
Sep 09 20:38:27.190652 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:38:27.190626    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7632cc6_320f_47fb_9dad_ecb76e4a94a7.slice/crio-747ea2c53d269dcd45966523a20f780f7a5bd52ece30a17b18b6b1494f2fe675 WatchSource:0}: Error finding container 747ea2c53d269dcd45966523a20f780f7a5bd52ece30a17b18b6b1494f2fe675: Status 404 returned error can't find the container with id 747ea2c53d269dcd45966523a20f780f7a5bd52ece30a17b18b6b1494f2fe675
Sep 09 20:38:28.127889 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:28.127849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" event={"ID":"a7632cc6-320f-47fb-9dad-ecb76e4a94a7","Type":"ContainerStarted","Data":"a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb"}
Sep 09 20:38:28.128356 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:28.127905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" event={"ID":"a7632cc6-320f-47fb-9dad-ecb76e4a94a7","Type":"ContainerStarted","Data":"747ea2c53d269dcd45966523a20f780f7a5bd52ece30a17b18b6b1494f2fe675"}
Sep 09 20:38:28.128356 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:28.127967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:38:28.144958 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:28.144905    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podStartSLOduration=2.144891407 podStartE2EDuration="2.144891407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:28.143253257 +0000 UTC m=+1624.117023319" watchObservedRunningTime="2026-09-09 20:38:28.144891407 +0000 UTC m=+1624.118661470"
Sep 09 20:38:34.138690 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:38:34.138661    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:41:24.516111 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:41:24.516077    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:41:24.518720 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:41:24.518688    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:45:51.307127 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.307093    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"]
Sep 09 20:45:51.307604 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.307347    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279" containerID="cri-o://263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084" gracePeriod=30
Sep 09 20:45:51.433317 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.433287    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"]
Sep 09 20:45:51.433542 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.433514    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container" containerID="cri-o://cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef" gracePeriod=30
Sep 09 20:45:51.433642 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.433554    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kube-rbac-proxy" containerID="cri-o://7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46" gracePeriod=30
Sep 09 20:45:51.537743 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.537707    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"]
Sep 09 20:45:51.541111 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.541091    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.543736 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.543712    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-326f5-kube-rbac-proxy-sar-config\""
Sep 09 20:45:51.543863 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.543741    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-326f5-predictor-serving-cert\""
Sep 09 20:45:51.549896 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.549875    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"]
Sep 09 20:45:51.738163 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.738131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.738163 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.738166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznxw\" (UniqueName: \"kubernetes.io/projected/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-api-access-zznxw\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.738376 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.738204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018573c1-e63e-49ed-9729-46b0d4836ce9-proxy-tls\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.838573 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.838544    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.838573 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.838576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznxw\" (UniqueName: \"kubernetes.io/projected/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-api-access-zznxw\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.838788 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.838613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018573c1-e63e-49ed-9729-46b0d4836ce9-proxy-tls\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.839231 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.839207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.840987 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.840957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018573c1-e63e-49ed-9729-46b0d4836ce9-proxy-tls\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.846319 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.846299    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznxw\" (UniqueName: \"kubernetes.io/projected/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-api-access-zznxw\") pod \"success-200-isvc-326f5-predictor-669c54dd8c-85sxw\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") " pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:51.852124 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:51.852098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:52.062314 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:52.062229    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerID="7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46" exitCode=2
Sep 09 20:45:52.062314 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:52.062290    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" event={"ID":"fd0dffb6-8f17-488c-93cf-5ec5c832153a","Type":"ContainerDied","Data":"7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46"}
Sep 09 20:45:52.121996 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:52.121974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"]
Sep 09 20:45:52.124012 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:45:52.123976    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018573c1_e63e_49ed_9729_46b0d4836ce9.slice/crio-bb35a61ed2bcfffcb5b417f637da45314f34538d9220e786f25682e6adc5f1c2 WatchSource:0}: Error finding container bb35a61ed2bcfffcb5b417f637da45314f34538d9220e786f25682e6adc5f1c2: Status 404 returned error can't find the container with id bb35a61ed2bcfffcb5b417f637da45314f34538d9220e786f25682e6adc5f1c2
Sep 09 20:45:53.068675 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:53.068639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" event={"ID":"018573c1-e63e-49ed-9729-46b0d4836ce9","Type":"ContainerStarted","Data":"f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e"}
Sep 09 20:45:53.069071 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:53.068704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" event={"ID":"018573c1-e63e-49ed-9729-46b0d4836ce9","Type":"ContainerStarted","Data":"8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520"}
Sep 09 20:45:53.069071 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:53.068731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" event={"ID":"018573c1-e63e-49ed-9729-46b0d4836ce9","Type":"ContainerStarted","Data":"bb35a61ed2bcfffcb5b417f637da45314f34538d9220e786f25682e6adc5f1c2"}
Sep 09 20:45:53.069071 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:53.068752    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:53.086073 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:53.086024    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podStartSLOduration=2.085993856 podStartE2EDuration="2.085993856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:53.085034971 +0000 UTC m=+2069.058805034" watchObservedRunningTime="2026-09-09 20:45:53.085993856 +0000 UTC m=+2069.059763919"
Sep 09 20:45:53.899144 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:53.899106    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:54.073688 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.073651    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:45:54.075238 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.075208    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 20:45:54.481546 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.481521    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:45:54.561969 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.561888    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls\") pod \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") "
Sep 09 20:45:54.561969 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.561949    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-rbac-proxy-sar-config\") pod \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") "
Sep 09 20:45:54.561969 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.561968    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7b5t\" (UniqueName: \"kubernetes.io/projected/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-api-access-p7b5t\") pod \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\" (UID: \"fd0dffb6-8f17-488c-93cf-5ec5c832153a\") "
Sep 09 20:45:54.562281 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.562258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd0dffb6-8f17-488c-93cf-5ec5c832153a" (UID: "fd0dffb6-8f17-488c-93cf-5ec5c832153a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:54.563987 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.563961    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd0dffb6-8f17-488c-93cf-5ec5c832153a" (UID: "fd0dffb6-8f17-488c-93cf-5ec5c832153a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:54.564111 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.564094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-api-access-p7b5t" (OuterVolumeSpecName: "kube-api-access-p7b5t") pod "fd0dffb6-8f17-488c-93cf-5ec5c832153a" (UID: "fd0dffb6-8f17-488c-93cf-5ec5c832153a"). InnerVolumeSpecName "kube-api-access-p7b5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:54.663044 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.663018    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:45:54.663044 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.663040    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7b5t\" (UniqueName: \"kubernetes.io/projected/fd0dffb6-8f17-488c-93cf-5ec5c832153a-kube-api-access-p7b5t\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:45:54.663185 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:54.663051    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd0dffb6-8f17-488c-93cf-5ec5c832153a-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:45:55.079033 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.078978    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerID="cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef" exitCode=0
Sep 09 20:45:55.079397 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.079066    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"
Sep 09 20:45:55.079397 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.079058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" event={"ID":"fd0dffb6-8f17-488c-93cf-5ec5c832153a","Type":"ContainerDied","Data":"cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef"}
Sep 09 20:45:55.079397 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.079242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq" event={"ID":"fd0dffb6-8f17-488c-93cf-5ec5c832153a","Type":"ContainerDied","Data":"bc9fa3dd83462cdc725d17b2249006607d821b6b8b18e97338fcd967fab9b65c"}
Sep 09 20:45:55.079397 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.079285    2562 scope.go:117] "RemoveContainer" containerID="7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46"
Sep 09 20:45:55.079616 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.079584    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 20:45:55.087821 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.087790    2562 scope.go:117] "RemoveContainer" containerID="cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef"
Sep 09 20:45:55.095207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.095177    2562 scope.go:117] "RemoveContainer" containerID="7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46"
Sep 09 20:45:55.095455 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:45:55.095436    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46\": container with ID starting with 7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46 not found: ID does not exist" containerID="7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46"
Sep 09 20:45:55.095510 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.095462    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46"} err="failed to get container status \"7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46\": rpc error: code = NotFound desc = could not find container \"7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46\": container with ID starting with 7724593a277cbd51c0bc93db95ef950dad5a16dfa1be54a01bca491eac234e46 not found: ID does not exist"
Sep 09 20:45:55.095510 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.095479    2562 scope.go:117] "RemoveContainer" containerID="cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef"
Sep 09 20:45:55.095716 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:45:55.095697    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef\": container with ID starting with cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef not found: ID does not exist" containerID="cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef"
Sep 09 20:45:55.095771 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.095726    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef"} err="failed to get container status \"cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef\": rpc error: code = NotFound desc = could not find container \"cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef\": container with ID starting with cfbcf9efa9413df6f94b9f63d93a0b727d40f33933745690f3c0a97aa48af3ef not found: ID does not exist"
Sep 09 20:45:55.099310 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.099289    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"]
Sep 09 20:45:55.102588 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:55.102566    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81279-predictor-bb8f78f5c-qlsqq"]
Sep 09 20:45:56.465367 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:56.465336    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" path="/var/lib/kubelet/pods/fd0dffb6-8f17-488c-93cf-5ec5c832153a/volumes"
Sep 09 20:45:58.899267 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:45:58.899224    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:00.084378 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:00.084348    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:46:00.084853 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:00.084828    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 20:46:03.899130 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:03.899089    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:03.899612 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:03.899195    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:46:08.898697 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:08.898656    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:10.085529 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:10.085488    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 20:46:13.899128 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:13.899087    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:18.898692 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:18.898647    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:20.085707 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:20.085658    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 20:46:21.455393 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:21.455369    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:46:21.598351 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:21.598265    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls\") pod \"4461df58-2188-41a6-90a5-0c6b8e77aee2\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") "
Sep 09 20:46:21.598523 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:21.598362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4461df58-2188-41a6-90a5-0c6b8e77aee2-openshift-service-ca-bundle\") pod \"4461df58-2188-41a6-90a5-0c6b8e77aee2\" (UID: \"4461df58-2188-41a6-90a5-0c6b8e77aee2\") "
Sep 09 20:46:21.598738 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:21.598705    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4461df58-2188-41a6-90a5-0c6b8e77aee2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4461df58-2188-41a6-90a5-0c6b8e77aee2" (UID: "4461df58-2188-41a6-90a5-0c6b8e77aee2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:21.600351 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:21.600331    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4461df58-2188-41a6-90a5-0c6b8e77aee2" (UID: "4461df58-2188-41a6-90a5-0c6b8e77aee2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:21.699906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:21.699870    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461df58-2188-41a6-90a5-0c6b8e77aee2-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:46:21.699906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:21.699896    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4461df58-2188-41a6-90a5-0c6b8e77aee2-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:46:22.208663 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.208624    2562 generic.go:356] "Generic (PLEG): container finished" podID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerID="263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084" exitCode=0
Sep 09 20:46:22.208925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.208683    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"
Sep 09 20:46:22.208925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.208709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" event={"ID":"4461df58-2188-41a6-90a5-0c6b8e77aee2","Type":"ContainerDied","Data":"263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084"}
Sep 09 20:46:22.208925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.208763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95" event={"ID":"4461df58-2188-41a6-90a5-0c6b8e77aee2","Type":"ContainerDied","Data":"31dad35ec538a1adca1f3a195ad043ac4d3c20a041d72d851471a58c2bea83d5"}
Sep 09 20:46:22.208925 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.208785    2562 scope.go:117] "RemoveContainer" containerID="263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084"
Sep 09 20:46:22.216774 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.216755    2562 scope.go:117] "RemoveContainer" containerID="263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084"
Sep 09 20:46:22.217063 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:46:22.217037    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084\": container with ID starting with 263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084 not found: ID does not exist" containerID="263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084"
Sep 09 20:46:22.217170 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.217068    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084"} err="failed to get container status \"263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084\": rpc error: code = NotFound desc = could not find container \"263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084\": container with ID starting with 263d570360e3d1ed7d26ee5f7eaa6b5ef94155cae592f1bc38ae7fe10185a084 not found: ID does not exist"
Sep 09 20:46:22.231668 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.231646    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"]
Sep 09 20:46:22.235935 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.235915    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-81279-7ff94b5fb-vhq95"]
Sep 09 20:46:22.465710 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:22.465634    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" path="/var/lib/kubelet/pods/4461df58-2188-41a6-90a5-0c6b8e77aee2/volumes"
Sep 09 20:46:24.542352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:24.542320    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:46:24.546607 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:24.546576    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:46:30.084874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:30.084836    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 20:46:40.085179 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:40.085146    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:46:41.167442 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.167412    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"]
Sep 09 20:46:41.167806 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.167624    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88" containerID="cri-o://a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb" gracePeriod=30
Sep 09 20:46:41.266580 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.266549    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"]
Sep 09 20:46:41.266844 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.266806    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" containerID="cri-o://4add40fd9d92712acda5dc68daa549ae263e48e860e4d17fc5e548ad16451a2b" gracePeriod=30
Sep 09 20:46:41.266932 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.266836    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kube-rbac-proxy" containerID="cri-o://1f8fe6cd1ebb6c0c9e078f33326a536831a1c41a5514e9974160d125b32fec8d" gracePeriod=30
Sep 09 20:46:41.405951 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.405919    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"]
Sep 09 20:46:41.406295 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406281    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kube-rbac-proxy"
Sep 09 20:46:41.406341 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406297    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kube-rbac-proxy"
Sep 09 20:46:41.406341 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406314    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279"
Sep 09 20:46:41.406341 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406320    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279"
Sep 09 20:46:41.406341 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406330    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container"
Sep 09 20:46:41.406341 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406336    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container"
Sep 09 20:46:41.406490 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406386    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kserve-container"
Sep 09 20:46:41.406490 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406396    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4461df58-2188-41a6-90a5-0c6b8e77aee2" containerName="ensemble-graph-81279"
Sep 09 20:46:41.406490 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.406406    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0dffb6-8f17-488c-93cf-5ec5c832153a" containerName="kube-rbac-proxy"
Sep 09 20:46:41.409423 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.409405    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.412401 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.412378    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6a9b1-predictor-serving-cert\""
Sep 09 20:46:41.412527 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.412442    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6a9b1-kube-rbac-proxy-sar-config\""
Sep 09 20:46:41.421583 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.421530    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"]
Sep 09 20:46:41.558179 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.558109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkfb\" (UniqueName: \"kubernetes.io/projected/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-api-access-xfkfb\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.558351 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.558226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3442c5bd-a860-42e5-894f-d5ef7edf7da8-proxy-tls\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.558351 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.558302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.658718 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.658685    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkfb\" (UniqueName: \"kubernetes.io/projected/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-api-access-xfkfb\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.658906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.658740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3442c5bd-a860-42e5-894f-d5ef7edf7da8-proxy-tls\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.658906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.658774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.659439 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.659416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.661050 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.661032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3442c5bd-a860-42e5-894f-d5ef7edf7da8-proxy-tls\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.666378 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.666357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkfb\" (UniqueName: \"kubernetes.io/projected/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-api-access-xfkfb\") pod \"success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") " pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.719985 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.719912    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:41.988639 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:41.988613    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"]
Sep 09 20:46:41.991107 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:46:41.991078    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3442c5bd_a860_42e5_894f_d5ef7edf7da8.slice/crio-a71d343b8cb73d30918d61f1a1faf766c2cd4061f6fb4f3b0968f38f80a8c464 WatchSource:0}: Error finding container a71d343b8cb73d30918d61f1a1faf766c2cd4061f6fb4f3b0968f38f80a8c464: Status 404 returned error can't find the container with id a71d343b8cb73d30918d61f1a1faf766c2cd4061f6fb4f3b0968f38f80a8c464
Sep 09 20:46:42.300254 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.300160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" event={"ID":"3442c5bd-a860-42e5-894f-d5ef7edf7da8","Type":"ContainerStarted","Data":"067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d"}
Sep 09 20:46:42.300254 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.300232    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:42.300736 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.300367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:42.300736 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.300417    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" event={"ID":"3442c5bd-a860-42e5-894f-d5ef7edf7da8","Type":"ContainerStarted","Data":"53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2"}
Sep 09 20:46:42.300736 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.300449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" event={"ID":"3442c5bd-a860-42e5-894f-d5ef7edf7da8","Type":"ContainerStarted","Data":"a71d343b8cb73d30918d61f1a1faf766c2cd4061f6fb4f3b0968f38f80a8c464"}
Sep 09 20:46:42.301750 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.301722    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 20:46:42.302223 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.302204    2562 generic.go:356] "Generic (PLEG): container finished" podID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerID="1f8fe6cd1ebb6c0c9e078f33326a536831a1c41a5514e9974160d125b32fec8d" exitCode=2
Sep 09 20:46:42.302281 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.302260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" event={"ID":"bb665308-c203-4bd9-aedc-9d9078d099ef","Type":"ContainerDied","Data":"1f8fe6cd1ebb6c0c9e078f33326a536831a1c41a5514e9974160d125b32fec8d"}
Sep 09 20:46:42.318371 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:42.318315    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podStartSLOduration=1.318297241 podStartE2EDuration="1.318297241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:42.316605156 +0000 UTC m=+2118.290375232" watchObservedRunningTime="2026-09-09 20:46:42.318297241 +0000 UTC m=+2118.292067303"
Sep 09 20:46:43.306551 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:43.306504    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 20:46:43.848688 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:43.848645    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 09 20:46:43.853561 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:43.853535    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 20:46:44.136932 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.136893    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:44.314303 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.314269    2562 generic.go:356] "Generic (PLEG): container finished" podID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerID="4add40fd9d92712acda5dc68daa549ae263e48e860e4d17fc5e548ad16451a2b" exitCode=0
Sep 09 20:46:44.314782 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.314351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" event={"ID":"bb665308-c203-4bd9-aedc-9d9078d099ef","Type":"ContainerDied","Data":"4add40fd9d92712acda5dc68daa549ae263e48e860e4d17fc5e548ad16451a2b"}
Sep 09 20:46:44.412473 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.412447    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:46:44.583153 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.583118    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8p5v\" (UniqueName: \"kubernetes.io/projected/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-api-access-d8p5v\") pod \"bb665308-c203-4bd9-aedc-9d9078d099ef\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") "
Sep 09 20:46:44.583348 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.583160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb665308-c203-4bd9-aedc-9d9078d099ef-proxy-tls\") pod \"bb665308-c203-4bd9-aedc-9d9078d099ef\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") "
Sep 09 20:46:44.583348 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.583287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-rbac-proxy-sar-config\") pod \"bb665308-c203-4bd9-aedc-9d9078d099ef\" (UID: \"bb665308-c203-4bd9-aedc-9d9078d099ef\") "
Sep 09 20:46:44.583642 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.583619    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb665308-c203-4bd9-aedc-9d9078d099ef" (UID: "bb665308-c203-4bd9-aedc-9d9078d099ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:44.585333 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.585307    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-api-access-d8p5v" (OuterVolumeSpecName: "kube-api-access-d8p5v") pod "bb665308-c203-4bd9-aedc-9d9078d099ef" (UID: "bb665308-c203-4bd9-aedc-9d9078d099ef"). InnerVolumeSpecName "kube-api-access-d8p5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:44.585425 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.585343    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb665308-c203-4bd9-aedc-9d9078d099ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb665308-c203-4bd9-aedc-9d9078d099ef" (UID: "bb665308-c203-4bd9-aedc-9d9078d099ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:44.684765 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.684669    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:46:44.684765 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.684703    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8p5v\" (UniqueName: \"kubernetes.io/projected/bb665308-c203-4bd9-aedc-9d9078d099ef-kube-api-access-d8p5v\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:46:44.684765 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:44.684718    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb665308-c203-4bd9-aedc-9d9078d099ef-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:46:45.320805 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:45.320775    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"
Sep 09 20:46:45.321233 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:45.320784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg" event={"ID":"bb665308-c203-4bd9-aedc-9d9078d099ef","Type":"ContainerDied","Data":"6d25e201c648f6c5ca4e5becb78e2fd8f03f3470d6a4ab6c7c7ccffc6d12ec22"}
Sep 09 20:46:45.321233 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:45.320927    2562 scope.go:117] "RemoveContainer" containerID="1f8fe6cd1ebb6c0c9e078f33326a536831a1c41a5514e9974160d125b32fec8d"
Sep 09 20:46:45.329788 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:45.329769    2562 scope.go:117] "RemoveContainer" containerID="4add40fd9d92712acda5dc68daa549ae263e48e860e4d17fc5e548ad16451a2b"
Sep 09 20:46:45.342818 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:45.342155    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"]
Sep 09 20:46:45.347852 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:45.346558    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-28e88-predictor-779f848669-jfwgg"]
Sep 09 20:46:46.465936 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:46.465901    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" path="/var/lib/kubelet/pods/bb665308-c203-4bd9-aedc-9d9078d099ef/volumes"
Sep 09 20:46:48.310924 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:48.310893    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:46:48.311531 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:48.311496    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 20:46:49.136846 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:49.136804    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:51.554784 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.554751    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"]
Sep 09 20:46:51.555176 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.555094    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container"
Sep 09 20:46:51.555176 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.555106    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container"
Sep 09 20:46:51.555176 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.555121    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kube-rbac-proxy"
Sep 09 20:46:51.555176 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.555127    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kube-rbac-proxy"
Sep 09 20:46:51.555310 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.555182    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kserve-container"
Sep 09 20:46:51.555310 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.555189    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb665308-c203-4bd9-aedc-9d9078d099ef" containerName="kube-rbac-proxy"
Sep 09 20:46:51.560442 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.560422    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:51.563069 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.563049    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-326f5-kube-rbac-proxy-sar-config\""
Sep 09 20:46:51.563069 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.563062    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-326f5-serving-cert\""
Sep 09 20:46:51.568380 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.568359    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"]
Sep 09 20:46:51.640084 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.640050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5abd73e6-2822-475e-ad09-134541e15617-proxy-tls\") pod \"splitter-graph-326f5-768586467d-gw8kd\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") " pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:51.640229 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.640127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abd73e6-2822-475e-ad09-134541e15617-openshift-service-ca-bundle\") pod \"splitter-graph-326f5-768586467d-gw8kd\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") " pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:51.740866 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.740834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abd73e6-2822-475e-ad09-134541e15617-openshift-service-ca-bundle\") pod \"splitter-graph-326f5-768586467d-gw8kd\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") " pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:51.741027 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.740891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5abd73e6-2822-475e-ad09-134541e15617-proxy-tls\") pod \"splitter-graph-326f5-768586467d-gw8kd\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") " pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:51.741520 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.741489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abd73e6-2822-475e-ad09-134541e15617-openshift-service-ca-bundle\") pod \"splitter-graph-326f5-768586467d-gw8kd\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") " pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:51.743263 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.743240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5abd73e6-2822-475e-ad09-134541e15617-proxy-tls\") pod \"splitter-graph-326f5-768586467d-gw8kd\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") " pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:51.871467 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:51.871397    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:52.139946 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:52.139923    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"]
Sep 09 20:46:52.142274 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:46:52.142245    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abd73e6_2822_475e_ad09_134541e15617.slice/crio-487b48d7d0f79210b72466dbd0c2a0c91fcd9984d8814301fb5c14bf6be8f301 WatchSource:0}: Error finding container 487b48d7d0f79210b72466dbd0c2a0c91fcd9984d8814301fb5c14bf6be8f301: Status 404 returned error can't find the container with id 487b48d7d0f79210b72466dbd0c2a0c91fcd9984d8814301fb5c14bf6be8f301
Sep 09 20:46:52.355468 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:52.355431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" event={"ID":"5abd73e6-2822-475e-ad09-134541e15617","Type":"ContainerStarted","Data":"bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247"}
Sep 09 20:46:52.355642 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:52.355486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" event={"ID":"5abd73e6-2822-475e-ad09-134541e15617","Type":"ContainerStarted","Data":"487b48d7d0f79210b72466dbd0c2a0c91fcd9984d8814301fb5c14bf6be8f301"}
Sep 09 20:46:52.355642 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:52.355513    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:52.371217 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:52.371168    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podStartSLOduration=1.371151359 podStartE2EDuration="1.371151359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:52.369669568 +0000 UTC m=+2128.343439630" watchObservedRunningTime="2026-09-09 20:46:52.371151359 +0000 UTC m=+2128.344921451"
Sep 09 20:46:54.136310 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:54.136274    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:54.136782 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:54.136405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:46:58.312405 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:58.312359    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 20:46:58.364841 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:58.364808    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:46:59.137122 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:46:59.137075    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:01.700723 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.700693    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"]
Sep 09 20:47:01.701131 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.700889    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5" containerID="cri-o://bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247" gracePeriod=30
Sep 09 20:47:01.797877 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.797830    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"]
Sep 09 20:47:01.798183 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.798156    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container" containerID="cri-o://8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520" gracePeriod=30
Sep 09 20:47:01.798296 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.798267    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kube-rbac-proxy" containerID="cri-o://f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e" gracePeriod=30
Sep 09 20:47:01.917578 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.917552    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"]
Sep 09 20:47:01.921040 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.921024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:01.923531 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.923509    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7dad6-kube-rbac-proxy-sar-config\""
Sep 09 20:47:01.923661 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.923640    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7dad6-predictor-serving-cert\""
Sep 09 20:47:01.930342 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:01.930318    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"]
Sep 09 20:47:02.032790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.032703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhzw\" (UniqueName: \"kubernetes.io/projected/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-api-access-drhzw\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.032790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.032748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-proxy-tls\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.032790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.032784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.134188 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.134149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhzw\" (UniqueName: \"kubernetes.io/projected/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-api-access-drhzw\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.134188 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.134192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-proxy-tls\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.134459 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.134217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.134868 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.134843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.136666 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.136645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-proxy-tls\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.142292 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.142270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhzw\" (UniqueName: \"kubernetes.io/projected/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-api-access-drhzw\") pod \"success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") " pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.231472 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.231428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:02.401411 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.401381    2562 generic.go:356] "Generic (PLEG): container finished" podID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerID="f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e" exitCode=2
Sep 09 20:47:02.401575 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.401450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" event={"ID":"018573c1-e63e-49ed-9729-46b0d4836ce9","Type":"ContainerDied","Data":"f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e"}
Sep 09 20:47:02.502196 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:02.502170    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"]
Sep 09 20:47:02.504258 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:47:02.504236    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f46f9a_7a7f_4b1f_ad91_8d0035a8c2c0.slice/crio-261eb786921d831f66e6eae6f972c3f17e946c143e25af77431b07dbb598c272 WatchSource:0}: Error finding container 261eb786921d831f66e6eae6f972c3f17e946c143e25af77431b07dbb598c272: Status 404 returned error can't find the container with id 261eb786921d831f66e6eae6f972c3f17e946c143e25af77431b07dbb598c272
Sep 09 20:47:03.364097 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:03.364053    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:03.410343 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:03.410297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" event={"ID":"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0","Type":"ContainerStarted","Data":"9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3"}
Sep 09 20:47:03.410528 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:03.410404    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:03.410528 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:03.410439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" event={"ID":"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0","Type":"ContainerStarted","Data":"7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e"}
Sep 09 20:47:03.410528 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:03.410474    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" event={"ID":"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0","Type":"ContainerStarted","Data":"261eb786921d831f66e6eae6f972c3f17e946c143e25af77431b07dbb598c272"}
Sep 09 20:47:03.428167 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:03.428114    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podStartSLOduration=2.4280984 podStartE2EDuration="2.4280984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:03.425879104 +0000 UTC m=+2139.399649166" watchObservedRunningTime="2026-09-09 20:47:03.4280984 +0000 UTC m=+2139.401868463"
Sep 09 20:47:04.136571 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:04.136535    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:04.416242 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:04.416144    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:04.417757 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:04.417724    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 20:47:05.154766 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.154741    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:47:05.261828 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.261740    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018573c1-e63e-49ed-9729-46b0d4836ce9-proxy-tls\") pod \"018573c1-e63e-49ed-9729-46b0d4836ce9\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") "
Sep 09 20:47:05.261828 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.261815    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zznxw\" (UniqueName: \"kubernetes.io/projected/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-api-access-zznxw\") pod \"018573c1-e63e-49ed-9729-46b0d4836ce9\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") "
Sep 09 20:47:05.262055 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.261867    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-rbac-proxy-sar-config\") pod \"018573c1-e63e-49ed-9729-46b0d4836ce9\" (UID: \"018573c1-e63e-49ed-9729-46b0d4836ce9\") "
Sep 09 20:47:05.262299 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.262264    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "018573c1-e63e-49ed-9729-46b0d4836ce9" (UID: "018573c1-e63e-49ed-9729-46b0d4836ce9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:05.263903 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.263879    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-api-access-zznxw" (OuterVolumeSpecName: "kube-api-access-zznxw") pod "018573c1-e63e-49ed-9729-46b0d4836ce9" (UID: "018573c1-e63e-49ed-9729-46b0d4836ce9"). InnerVolumeSpecName "kube-api-access-zznxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:05.263997 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.263947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018573c1-e63e-49ed-9729-46b0d4836ce9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "018573c1-e63e-49ed-9729-46b0d4836ce9" (UID: "018573c1-e63e-49ed-9729-46b0d4836ce9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:05.362958 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.362923    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zznxw\" (UniqueName: \"kubernetes.io/projected/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-api-access-zznxw\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:47:05.362958 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.362952    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018573c1-e63e-49ed-9729-46b0d4836ce9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:47:05.363213 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.362980    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018573c1-e63e-49ed-9729-46b0d4836ce9-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:47:05.422538 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.422501    2562 generic.go:356] "Generic (PLEG): container finished" podID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerID="8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520" exitCode=0
Sep 09 20:47:05.422918 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.422565    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"
Sep 09 20:47:05.422918 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.422578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" event={"ID":"018573c1-e63e-49ed-9729-46b0d4836ce9","Type":"ContainerDied","Data":"8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520"}
Sep 09 20:47:05.422918 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.422671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" event={"ID":"018573c1-e63e-49ed-9729-46b0d4836ce9","Type":"ContainerDied","Data":"bb35a61ed2bcfffcb5b417f637da45314f34538d9220e786f25682e6adc5f1c2"}
Sep 09 20:47:05.422918 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.422704    2562 scope.go:117] "RemoveContainer" containerID="f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e"
Sep 09 20:47:05.423267 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.423234    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 20:47:05.431613 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.431595    2562 scope.go:117] "RemoveContainer" containerID="8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520"
Sep 09 20:47:05.439695 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.439675    2562 scope.go:117] "RemoveContainer" containerID="f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e"
Sep 09 20:47:05.439967 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:47:05.439946    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e\": container with ID starting with f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e not found: ID does not exist" containerID="f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e"
Sep 09 20:47:05.440078 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.439975    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e"} err="failed to get container status \"f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e\": rpc error: code = NotFound desc = could not find container \"f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e\": container with ID starting with f4ec860e1823c57554c355922a0601d9869f74c5a29a108c240fb875d0023c7e not found: ID does not exist"
Sep 09 20:47:05.440078 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.439992    2562 scope.go:117] "RemoveContainer" containerID="8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520"
Sep 09 20:47:05.440258 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:47:05.440238    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520\": container with ID starting with 8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520 not found: ID does not exist" containerID="8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520"
Sep 09 20:47:05.440314 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.440268    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520"} err="failed to get container status \"8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520\": rpc error: code = NotFound desc = could not find container \"8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520\": container with ID starting with 8239918f27ee42d68c5edfbe87264f5d1071c79cc83336f11581ead14260f520 not found: ID does not exist"
Sep 09 20:47:05.444037 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.444014    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"]
Sep 09 20:47:05.447336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:05.447310    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw"]
Sep 09 20:47:06.081182 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:06.081136    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-326f5-predictor-669c54dd8c-85sxw" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:47:06.465578 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:06.465547    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" path="/var/lib/kubelet/pods/018573c1-e63e-49ed-9729-46b0d4836ce9/volumes"
Sep 09 20:47:08.311508 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:08.311465    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 20:47:08.363614 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:08.363578    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:09.136779 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:09.136736    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:10.427736 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:10.427704    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:47:10.428198 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:10.428164    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 20:47:11.317293 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.317270    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:47:11.412709 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.412679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-openshift-service-ca-bundle\") pod \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") "
Sep 09 20:47:11.412879 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.412716    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-proxy-tls\") pod \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\" (UID: \"a7632cc6-320f-47fb-9dad-ecb76e4a94a7\") "
Sep 09 20:47:11.413147 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.413117    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a7632cc6-320f-47fb-9dad-ecb76e4a94a7" (UID: "a7632cc6-320f-47fb-9dad-ecb76e4a94a7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:11.414754 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.414713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7632cc6-320f-47fb-9dad-ecb76e4a94a7" (UID: "a7632cc6-320f-47fb-9dad-ecb76e4a94a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:11.451921 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.451855    2562 generic.go:356] "Generic (PLEG): container finished" podID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerID="a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb" exitCode=0
Sep 09 20:47:11.451921 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.451908    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"
Sep 09 20:47:11.452352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.451936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" event={"ID":"a7632cc6-320f-47fb-9dad-ecb76e4a94a7","Type":"ContainerDied","Data":"a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb"}
Sep 09 20:47:11.452352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.452020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8" event={"ID":"a7632cc6-320f-47fb-9dad-ecb76e4a94a7","Type":"ContainerDied","Data":"747ea2c53d269dcd45966523a20f780f7a5bd52ece30a17b18b6b1494f2fe675"}
Sep 09 20:47:11.452352 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.452043    2562 scope.go:117] "RemoveContainer" containerID="a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb"
Sep 09 20:47:11.460253 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.460237    2562 scope.go:117] "RemoveContainer" containerID="a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb"
Sep 09 20:47:11.460538 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:47:11.460517    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb\": container with ID starting with a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb not found: ID does not exist" containerID="a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb"
Sep 09 20:47:11.460590 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.460546    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb"} err="failed to get container status \"a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb\": rpc error: code = NotFound desc = could not find container \"a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb\": container with ID starting with a94698e62698fe5922abd5b3895e84b232b73359aafc603ff84f1cff52f9a0fb not found: ID does not exist"
Sep 09 20:47:11.471611 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.471592    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"]
Sep 09 20:47:11.477507 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.477486    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-28e88-f549854fc-7qtn8"]
Sep 09 20:47:11.513219 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.513195    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:47:11.513219 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:11.513213    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7632cc6-320f-47fb-9dad-ecb76e4a94a7-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:47:12.465902 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:12.465869    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" path="/var/lib/kubelet/pods/a7632cc6-320f-47fb-9dad-ecb76e4a94a7/volumes"
Sep 09 20:47:13.363813 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:13.363778    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:13.363985 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:13.363880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:47:18.311502 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:18.311459    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 20:47:18.363626 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:18.363590    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:20.428590 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:20.428552    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 20:47:23.363345 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:23.363299    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:28.312173 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:28.312135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 20:47:28.363798 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:28.363762    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:30.428257 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:30.428216    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 20:47:31.864886 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:31.864863    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:47:31.977573 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:31.977500    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5abd73e6-2822-475e-ad09-134541e15617-proxy-tls\") pod \"5abd73e6-2822-475e-ad09-134541e15617\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") "
Sep 09 20:47:31.977573 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:31.977546    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abd73e6-2822-475e-ad09-134541e15617-openshift-service-ca-bundle\") pod \"5abd73e6-2822-475e-ad09-134541e15617\" (UID: \"5abd73e6-2822-475e-ad09-134541e15617\") "
Sep 09 20:47:31.977906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:31.977885    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abd73e6-2822-475e-ad09-134541e15617-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5abd73e6-2822-475e-ad09-134541e15617" (UID: "5abd73e6-2822-475e-ad09-134541e15617"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:31.979452 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:31.979426    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abd73e6-2822-475e-ad09-134541e15617-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5abd73e6-2822-475e-ad09-134541e15617" (UID: "5abd73e6-2822-475e-ad09-134541e15617"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:32.078558 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.078526    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5abd73e6-2822-475e-ad09-134541e15617-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:47:32.078558 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.078557    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abd73e6-2822-475e-ad09-134541e15617-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:47:32.545077 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.545045    2562 generic.go:356] "Generic (PLEG): container finished" podID="5abd73e6-2822-475e-ad09-134541e15617" containerID="bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247" exitCode=137
Sep 09 20:47:32.545218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.545084    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"
Sep 09 20:47:32.545218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.545128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" event={"ID":"5abd73e6-2822-475e-ad09-134541e15617","Type":"ContainerDied","Data":"bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247"}
Sep 09 20:47:32.545218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.545180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd" event={"ID":"5abd73e6-2822-475e-ad09-134541e15617","Type":"ContainerDied","Data":"487b48d7d0f79210b72466dbd0c2a0c91fcd9984d8814301fb5c14bf6be8f301"}
Sep 09 20:47:32.545218 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.545203    2562 scope.go:117] "RemoveContainer" containerID="bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247"
Sep 09 20:47:32.553211 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.553193    2562 scope.go:117] "RemoveContainer" containerID="bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247"
Sep 09 20:47:32.553463 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:47:32.553437    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247\": container with ID starting with bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247 not found: ID does not exist" containerID="bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247"
Sep 09 20:47:32.553526 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.553470    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247"} err="failed to get container status \"bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247\": rpc error: code = NotFound desc = could not find container \"bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247\": container with ID starting with bca66aff800d54c3e3805f89fce470232f25e99347ee29ed0173568cf66b1247 not found: ID does not exist"
Sep 09 20:47:32.562629 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.562604    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"]
Sep 09 20:47:32.566612 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:32.566592    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-326f5-768586467d-gw8kd"]
Sep 09 20:47:34.465186 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:34.465156    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5abd73e6-2822-475e-ad09-134541e15617" path="/var/lib/kubelet/pods/5abd73e6-2822-475e-ad09-134541e15617/volumes"
Sep 09 20:47:40.428332 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:40.428299    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 20:47:41.452362 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452330    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"]
Sep 09 20:47:41.452709 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452673    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5"
Sep 09 20:47:41.452709 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452685    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5"
Sep 09 20:47:41.452709 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452696    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88"
Sep 09 20:47:41.452709 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452702    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88"
Sep 09 20:47:41.452709 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452710    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kube-rbac-proxy"
Sep 09 20:47:41.452874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452716    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kube-rbac-proxy"
Sep 09 20:47:41.452874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452725    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container"
Sep 09 20:47:41.452874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452733    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container"
Sep 09 20:47:41.452874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452797    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kserve-container"
Sep 09 20:47:41.452874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452806    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7632cc6-320f-47fb-9dad-ecb76e4a94a7" containerName="sequence-graph-28e88"
Sep 09 20:47:41.452874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452815    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="018573c1-e63e-49ed-9729-46b0d4836ce9" containerName="kube-rbac-proxy"
Sep 09 20:47:41.452874 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.452825    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5abd73e6-2822-475e-ad09-134541e15617" containerName="splitter-graph-326f5"
Sep 09 20:47:41.455867 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.455850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:41.458377 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.458349    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6a9b1-kube-rbac-proxy-sar-config\""
Sep 09 20:47:41.458529 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.458395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6a9b1-serving-cert\""
Sep 09 20:47:41.463864 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.463841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"]
Sep 09 20:47:41.547790 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.547764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-openshift-service-ca-bundle\") pod \"switch-graph-6a9b1-6d87596d7f-pdg8z\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") " pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:41.547906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.547815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls\") pod \"switch-graph-6a9b1-6d87596d7f-pdg8z\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") " pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:41.648239 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.648203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-openshift-service-ca-bundle\") pod \"switch-graph-6a9b1-6d87596d7f-pdg8z\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") " pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:41.648405 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.648276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls\") pod \"switch-graph-6a9b1-6d87596d7f-pdg8z\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") " pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:41.648472 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:47:41.648413    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-6a9b1-serving-cert: secret "switch-graph-6a9b1-serving-cert" not found
Sep 09 20:47:41.648472 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:47:41.648471    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls podName:768f6010-5ce0-40ad-ab30-a88b9d25b1aa nodeName:}" failed. No retries permitted until 2026-09-09 20:47:42.148454821 +0000 UTC m=+2178.122224878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls") pod "switch-graph-6a9b1-6d87596d7f-pdg8z" (UID: "768f6010-5ce0-40ad-ab30-a88b9d25b1aa") : secret "switch-graph-6a9b1-serving-cert" not found
Sep 09 20:47:41.648868 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:41.648848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-openshift-service-ca-bundle\") pod \"switch-graph-6a9b1-6d87596d7f-pdg8z\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") " pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:42.150966 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:42.150933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls\") pod \"switch-graph-6a9b1-6d87596d7f-pdg8z\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") " pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:42.153384 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:42.153364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls\") pod \"switch-graph-6a9b1-6d87596d7f-pdg8z\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") " pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:42.366735 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:42.366695    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:42.636288 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:42.636267    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"]
Sep 09 20:47:42.638361 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:47:42.638330    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768f6010_5ce0_40ad_ab30_a88b9d25b1aa.slice/crio-d83a35c085ac692c565d078f2627b66f4eea50750c10b29b389e6501633cac8e WatchSource:0}: Error finding container d83a35c085ac692c565d078f2627b66f4eea50750c10b29b389e6501633cac8e: Status 404 returned error can't find the container with id d83a35c085ac692c565d078f2627b66f4eea50750c10b29b389e6501633cac8e
Sep 09 20:47:43.595141 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:43.595108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" event={"ID":"768f6010-5ce0-40ad-ab30-a88b9d25b1aa","Type":"ContainerStarted","Data":"8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67"}
Sep 09 20:47:43.595309 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:43.595158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" event={"ID":"768f6010-5ce0-40ad-ab30-a88b9d25b1aa","Type":"ContainerStarted","Data":"d83a35c085ac692c565d078f2627b66f4eea50750c10b29b389e6501633cac8e"}
Sep 09 20:47:43.595309 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:43.595210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:43.611650 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:43.611602    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podStartSLOduration=2.6115874359999998 podStartE2EDuration="2.611587436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:43.610575206 +0000 UTC m=+2179.584345270" watchObservedRunningTime="2026-09-09 20:47:43.611587436 +0000 UTC m=+2179.585357500"
Sep 09 20:47:49.605207 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:49.605177    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 20:47:50.428592 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:47:50.428561    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:48:01.916521 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:01.916482    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"]
Sep 09 20:48:01.919624 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:01.919606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:01.922106 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:01.922083    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7dad6-kube-rbac-proxy-sar-config\""
Sep 09 20:48:01.922212 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:01.922085    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7dad6-serving-cert\""
Sep 09 20:48:01.927628 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:01.927607    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"]
Sep 09 20:48:02.005167 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.005133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ba32cd-ef95-459a-81b8-e985dd913c8f-openshift-service-ca-bundle\") pod \"splitter-graph-7dad6-75dfbcc7d7-nd9vt\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") " pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.005340 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.005179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ba32cd-ef95-459a-81b8-e985dd913c8f-proxy-tls\") pod \"splitter-graph-7dad6-75dfbcc7d7-nd9vt\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") " pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.106129 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.106097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ba32cd-ef95-459a-81b8-e985dd913c8f-openshift-service-ca-bundle\") pod \"splitter-graph-7dad6-75dfbcc7d7-nd9vt\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") " pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.106129 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.106143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ba32cd-ef95-459a-81b8-e985dd913c8f-proxy-tls\") pod \"splitter-graph-7dad6-75dfbcc7d7-nd9vt\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") " pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.106738 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.106708    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ba32cd-ef95-459a-81b8-e985dd913c8f-openshift-service-ca-bundle\") pod \"splitter-graph-7dad6-75dfbcc7d7-nd9vt\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") " pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.108528 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.108510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ba32cd-ef95-459a-81b8-e985dd913c8f-proxy-tls\") pod \"splitter-graph-7dad6-75dfbcc7d7-nd9vt\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") " pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.230336 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.230241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.500173 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.500147    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"]
Sep 09 20:48:02.501992 ip-10-0-137-234 kubenswrapper[2562]: W0909 20:48:02.501965    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ba32cd_ef95_459a_81b8_e985dd913c8f.slice/crio-fbd774d26fbc529f7180820fc373b7d623241fcd9bcf7b2b54e195c942642a25 WatchSource:0}: Error finding container fbd774d26fbc529f7180820fc373b7d623241fcd9bcf7b2b54e195c942642a25: Status 404 returned error can't find the container with id fbd774d26fbc529f7180820fc373b7d623241fcd9bcf7b2b54e195c942642a25
Sep 09 20:48:02.683850 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.683811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" event={"ID":"e1ba32cd-ef95-459a-81b8-e985dd913c8f","Type":"ContainerStarted","Data":"49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e"}
Sep 09 20:48:02.684047 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.683870    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" event={"ID":"e1ba32cd-ef95-459a-81b8-e985dd913c8f","Type":"ContainerStarted","Data":"fbd774d26fbc529f7180820fc373b7d623241fcd9bcf7b2b54e195c942642a25"}
Sep 09 20:48:02.684047 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.683902    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:48:02.700493 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:02.700444    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podStartSLOduration=1.7004287329999999 podStartE2EDuration="1.700428733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:02.698679922 +0000 UTC m=+2198.672449985" watchObservedRunningTime="2026-09-09 20:48:02.700428733 +0000 UTC m=+2198.674198796"
Sep 09 20:48:08.694168 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:48:08.694135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:51:24.573819 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:51:24.573790    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:51:24.577390 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:51:24.577364    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:56:16.521823 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:16.521789    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"]
Sep 09 20:56:16.524333 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:16.521991    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6" containerID="cri-o://49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e" gracePeriod=30
Sep 09 20:56:16.635787 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:16.635756    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"]
Sep 09 20:56:16.636054 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:16.636019    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container" containerID="cri-o://7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e" gracePeriod=30
Sep 09 20:56:16.636054 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:16.636040    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kube-rbac-proxy" containerID="cri-o://9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3" gracePeriod=30
Sep 09 20:56:16.823710 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:16.823626    2562 generic.go:356] "Generic (PLEG): container finished" podID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerID="9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3" exitCode=2
Sep 09 20:56:16.823856 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:16.823701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" event={"ID":"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0","Type":"ContainerDied","Data":"9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3"}
Sep 09 20:56:18.692191 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:18.692155    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:19.377381 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.377359    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:56:19.478437 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.478371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drhzw\" (UniqueName: \"kubernetes.io/projected/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-api-access-drhzw\") pod \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") "
Sep 09 20:56:19.478437 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.478415    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-rbac-proxy-sar-config\") pod \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") "
Sep 09 20:56:19.478641 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.478484    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-proxy-tls\") pod \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\" (UID: \"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0\") "
Sep 09 20:56:19.478859 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.478831    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" (UID: "32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:19.480424 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.480395    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-api-access-drhzw" (OuterVolumeSpecName: "kube-api-access-drhzw") pod "32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" (UID: "32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0"). InnerVolumeSpecName "kube-api-access-drhzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:19.480508 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.480454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" (UID: "32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.579612 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.579584    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.579612 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.579608    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drhzw\" (UniqueName: \"kubernetes.io/projected/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-api-access-drhzw\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.579743 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.579618    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.837813 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.837735    2562 generic.go:356] "Generic (PLEG): container finished" podID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerID="7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e" exitCode=0
Sep 09 20:56:19.837813 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.837784    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"
Sep 09 20:56:19.838219 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.837821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" event={"ID":"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0","Type":"ContainerDied","Data":"7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e"}
Sep 09 20:56:19.838219 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.837880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9" event={"ID":"32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0","Type":"ContainerDied","Data":"261eb786921d831f66e6eae6f972c3f17e946c143e25af77431b07dbb598c272"}
Sep 09 20:56:19.838219 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.837906    2562 scope.go:117] "RemoveContainer" containerID="9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3"
Sep 09 20:56:19.846371 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.846358    2562 scope.go:117] "RemoveContainer" containerID="7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e"
Sep 09 20:56:19.859455 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.859430    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"]
Sep 09 20:56:19.859725 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.859710    2562 scope.go:117] "RemoveContainer" containerID="9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3"
Sep 09 20:56:19.859968 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:56:19.859949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3\": container with ID starting with 9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3 not found: ID does not exist" containerID="9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3"
Sep 09 20:56:19.860076 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.859977    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3"} err="failed to get container status \"9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3\": rpc error: code = NotFound desc = could not find container \"9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3\": container with ID starting with 9c46d6b74dea9bd77e6f842a6816de8a72ca4e55422b948beb1ff343e1b17ad3 not found: ID does not exist"
Sep 09 20:56:19.860076 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.860020    2562 scope.go:117] "RemoveContainer" containerID="7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e"
Sep 09 20:56:19.860262 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:56:19.860245    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e\": container with ID starting with 7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e not found: ID does not exist" containerID="7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e"
Sep 09 20:56:19.860302 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.860268    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e"} err="failed to get container status \"7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e\": rpc error: code = NotFound desc = could not find container \"7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e\": container with ID starting with 7a2535a20d427d67cc4943abc4002696853ea5d58c2710889ed3c337122ac05e not found: ID does not exist"
Sep 09 20:56:19.862732 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:19.862704    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7dad6-predictor-7c8fc894cf-46kb9"]
Sep 09 20:56:20.465862 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:20.465828    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" path="/var/lib/kubelet/pods/32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0/volumes"
Sep 09 20:56:23.692341 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:23.692303    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:24.600578 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:24.600538    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:56:24.605906 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:24.605880    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 20:56:28.692533 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:28.692493    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:28.692908 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:28.692595    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:56:33.692227 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:33.692189    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:38.692404 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:38.692363    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:43.692460 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:43.692419    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:46.707055 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.707023    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:56:46.787254 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.787194    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ba32cd-ef95-459a-81b8-e985dd913c8f-proxy-tls\") pod \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") "
Sep 09 20:56:46.787366 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.787267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ba32cd-ef95-459a-81b8-e985dd913c8f-openshift-service-ca-bundle\") pod \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\" (UID: \"e1ba32cd-ef95-459a-81b8-e985dd913c8f\") "
Sep 09 20:56:46.787585 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.787565    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ba32cd-ef95-459a-81b8-e985dd913c8f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e1ba32cd-ef95-459a-81b8-e985dd913c8f" (UID: "e1ba32cd-ef95-459a-81b8-e985dd913c8f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:46.789168 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.789149    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ba32cd-ef95-459a-81b8-e985dd913c8f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1ba32cd-ef95-459a-81b8-e985dd913c8f" (UID: "e1ba32cd-ef95-459a-81b8-e985dd913c8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:46.888064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.888040    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ba32cd-ef95-459a-81b8-e985dd913c8f-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:56:46.888064 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.888064    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ba32cd-ef95-459a-81b8-e985dd913c8f-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 20:56:46.955195 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.955170    2562 generic.go:356] "Generic (PLEG): container finished" podID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerID="49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e" exitCode=137
Sep 09 20:56:46.955312 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.955210    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"
Sep 09 20:56:46.955312 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.955261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" event={"ID":"e1ba32cd-ef95-459a-81b8-e985dd913c8f","Type":"ContainerDied","Data":"49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e"}
Sep 09 20:56:46.955312 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.955311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt" event={"ID":"e1ba32cd-ef95-459a-81b8-e985dd913c8f","Type":"ContainerDied","Data":"fbd774d26fbc529f7180820fc373b7d623241fcd9bcf7b2b54e195c942642a25"}
Sep 09 20:56:46.955423 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.955332    2562 scope.go:117] "RemoveContainer" containerID="49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e"
Sep 09 20:56:46.963733 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.963715    2562 scope.go:117] "RemoveContainer" containerID="49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e"
Sep 09 20:56:46.964031 ip-10-0-137-234 kubenswrapper[2562]: E0909 20:56:46.963989    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e\": container with ID starting with 49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e not found: ID does not exist" containerID="49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e"
Sep 09 20:56:46.964088 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.964040    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e"} err="failed to get container status \"49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e\": rpc error: code = NotFound desc = could not find container \"49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e\": container with ID starting with 49a681155be3a255d1aedbcbfb11b33e9da3a4dd15709f9ca197d01013b8da9e not found: ID does not exist"
Sep 09 20:56:46.975331 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.975308    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"]
Sep 09 20:56:46.977250 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:46.977229    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7dad6-75dfbcc7d7-nd9vt"]
Sep 09 20:56:48.465747 ip-10-0-137-234 kubenswrapper[2562]: I0909 20:56:48.465719    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" path="/var/lib/kubelet/pods/e1ba32cd-ef95-459a-81b8-e985dd913c8f/volumes"
Sep 09 21:01:24.625686 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:01:24.625652    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 21:01:24.639364 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:01:24.639340    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 21:04:00.966091 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:00.966050    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"]
Sep 09 21:04:00.966665 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:00.966291    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1" containerID="cri-o://8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67" gracePeriod=30
Sep 09 21:04:01.196983 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:01.196947    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"]
Sep 09 21:04:01.197233 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:01.197183    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container" containerID="cri-o://53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2" gracePeriod=30
Sep 09 21:04:01.197373 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:01.197280    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kube-rbac-proxy" containerID="cri-o://067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d" gracePeriod=30
Sep 09 21:04:01.792344 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:01.792311    2562 generic.go:356] "Generic (PLEG): container finished" podID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerID="067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d" exitCode=2
Sep 09 21:04:01.792497 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:01.792386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" event={"ID":"3442c5bd-a860-42e5-894f-d5ef7edf7da8","Type":"ContainerDied","Data":"067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d"}
Sep 09 21:04:03.307278 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:03.307230    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 09 21:04:04.243579 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.243555    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 21:04:04.402268 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.402231    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3442c5bd-a860-42e5-894f-d5ef7edf7da8-proxy-tls\") pod \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") "
Sep 09 21:04:04.402657 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.402307    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfkfb\" (UniqueName: \"kubernetes.io/projected/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-api-access-xfkfb\") pod \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") "
Sep 09 21:04:04.402657 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.402327    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-rbac-proxy-sar-config\") pod \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\" (UID: \"3442c5bd-a860-42e5-894f-d5ef7edf7da8\") "
Sep 09 21:04:04.402752 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.402714    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3442c5bd-a860-42e5-894f-d5ef7edf7da8" (UID: "3442c5bd-a860-42e5-894f-d5ef7edf7da8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:04.404355 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.404323    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3442c5bd-a860-42e5-894f-d5ef7edf7da8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3442c5bd-a860-42e5-894f-d5ef7edf7da8" (UID: "3442c5bd-a860-42e5-894f-d5ef7edf7da8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:04.404355 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.404328    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-api-access-xfkfb" (OuterVolumeSpecName: "kube-api-access-xfkfb") pod "3442c5bd-a860-42e5-894f-d5ef7edf7da8" (UID: "3442c5bd-a860-42e5-894f-d5ef7edf7da8"). InnerVolumeSpecName "kube-api-access-xfkfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:04.502872 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.502839    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfkfb\" (UniqueName: \"kubernetes.io/projected/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-api-access-xfkfb\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 21:04:04.502872 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.502869    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3442c5bd-a860-42e5-894f-d5ef7edf7da8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 21:04:04.502872 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.502879    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3442c5bd-a860-42e5-894f-d5ef7edf7da8-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 21:04:04.603062 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.603015    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:04.812617 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.812525    2562 generic.go:356] "Generic (PLEG): container finished" podID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerID="53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2" exitCode=0
Sep 09 21:04:04.812617 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.812585    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"
Sep 09 21:04:04.812617 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.812609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" event={"ID":"3442c5bd-a860-42e5-894f-d5ef7edf7da8","Type":"ContainerDied","Data":"53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2"}
Sep 09 21:04:04.812869 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.812691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw" event={"ID":"3442c5bd-a860-42e5-894f-d5ef7edf7da8","Type":"ContainerDied","Data":"a71d343b8cb73d30918d61f1a1faf766c2cd4061f6fb4f3b0968f38f80a8c464"}
Sep 09 21:04:04.812869 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.812740    2562 scope.go:117] "RemoveContainer" containerID="067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d"
Sep 09 21:04:04.820830 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.820810    2562 scope.go:117] "RemoveContainer" containerID="53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2"
Sep 09 21:04:04.828092 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.828073    2562 scope.go:117] "RemoveContainer" containerID="067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d"
Sep 09 21:04:04.828352 ip-10-0-137-234 kubenswrapper[2562]: E0909 21:04:04.828335    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d\": container with ID starting with 067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d not found: ID does not exist" containerID="067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d"
Sep 09 21:04:04.828441 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.828368    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d"} err="failed to get container status \"067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d\": rpc error: code = NotFound desc = could not find container \"067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d\": container with ID starting with 067b9e70189c7965243827becb89a3934171ebb98403770ad0d783c27f08c26d not found: ID does not exist"
Sep 09 21:04:04.828441 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.828384    2562 scope.go:117] "RemoveContainer" containerID="53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2"
Sep 09 21:04:04.828616 ip-10-0-137-234 kubenswrapper[2562]: E0909 21:04:04.828597    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2\": container with ID starting with 53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2 not found: ID does not exist" containerID="53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2"
Sep 09 21:04:04.828657 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.828625    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2"} err="failed to get container status \"53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2\": rpc error: code = NotFound desc = could not find container \"53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2\": container with ID starting with 53a94303e5d80f870e924954e49f587e439f4b6a1c93cfb4d98e4180d73496b2 not found: ID does not exist"
Sep 09 21:04:04.838619 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.838593    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"]
Sep 09 21:04:04.847605 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:04.847582    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a9b1-predictor-7bc68bf869-7z2fw"]
Sep 09 21:04:06.464947 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:06.464912    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" path="/var/lib/kubelet/pods/3442c5bd-a860-42e5-894f-d5ef7edf7da8/volumes"
Sep 09 21:04:09.603640 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:09.603597    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:14.603510 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:14.603470    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:14.603919 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:14.603584    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 21:04:18.418226 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:18.418196    2562 ???:1] "http: TLS handshake error from 10.0.142.76:52404: EOF"
Sep 09 21:04:18.422291 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:18.422268    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:19.259384 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:19.259354    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:19.603708 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:19.603613    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:20.061229 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:20.061182    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:20.850972 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:20.850943    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:21.635096 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:21.635067    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:22.417128 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:22.417088    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:23.209872 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:23.209836    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:23.997867 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:23.997833    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:24.603130 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:24.603096    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:24.802565 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:24.802538    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:25.587927 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:25.587886    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:26.355747 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:26.355711    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:27.155260 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:27.155230    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:27.948727 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:27.948698    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:28.738926 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:28.738896    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6a9b1-6d87596d7f-pdg8z_768f6010-5ce0-40ad-ab30-a88b9d25b1aa/switch-graph-6a9b1/0.log"
Sep 09 21:04:29.603621 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:29.603582    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:31.114477 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.114454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 21:04:31.218397 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.218366    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-openshift-service-ca-bundle\") pod \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") "
Sep 09 21:04:31.218571 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.218437    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls\") pod \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\" (UID: \"768f6010-5ce0-40ad-ab30-a88b9d25b1aa\") "
Sep 09 21:04:31.218760 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.218729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "768f6010-5ce0-40ad-ab30-a88b9d25b1aa" (UID: "768f6010-5ce0-40ad-ab30-a88b9d25b1aa"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:31.220533 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.220485    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "768f6010-5ce0-40ad-ab30-a88b9d25b1aa" (UID: "768f6010-5ce0-40ad-ab30-a88b9d25b1aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:31.319810 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.319780    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-openshift-service-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 21:04:31.319810 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.319807    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f6010-5ce0-40ad-ab30-a88b9d25b1aa-proxy-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Sep 09 21:04:31.925487 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.925450    2562 generic.go:356] "Generic (PLEG): container finished" podID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerID="8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67" exitCode=0
Sep 09 21:04:31.925663 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.925499    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"
Sep 09 21:04:31.925663 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.925514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" event={"ID":"768f6010-5ce0-40ad-ab30-a88b9d25b1aa","Type":"ContainerDied","Data":"8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67"}
Sep 09 21:04:31.925663 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.925570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z" event={"ID":"768f6010-5ce0-40ad-ab30-a88b9d25b1aa","Type":"ContainerDied","Data":"d83a35c085ac692c565d078f2627b66f4eea50750c10b29b389e6501633cac8e"}
Sep 09 21:04:31.925663 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.925599    2562 scope.go:117] "RemoveContainer" containerID="8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67"
Sep 09 21:04:31.933706 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.933688    2562 scope.go:117] "RemoveContainer" containerID="8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67"
Sep 09 21:04:31.933935 ip-10-0-137-234 kubenswrapper[2562]: E0909 21:04:31.933912    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67\": container with ID starting with 8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67 not found: ID does not exist" containerID="8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67"
Sep 09 21:04:31.933994 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.933948    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67"} err="failed to get container status \"8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67\": rpc error: code = NotFound desc = could not find container \"8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67\": container with ID starting with 8f75cdcf299bf38ab6c2c462343af9a55d96ebe8e56223bdf8d5046534375c67 not found: ID does not exist"
Sep 09 21:04:31.947800 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.947770    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"]
Sep 09 21:04:31.951136 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:31.951116    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6a9b1-6d87596d7f-pdg8z"]
Sep 09 21:04:32.465343 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:32.465310    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" path="/var/lib/kubelet/pods/768f6010-5ce0-40ad-ab30-a88b9d25b1aa/volumes"
Sep 09 21:04:33.773366 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773332    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cmgmq/must-gather-k52mz"]
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773655    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773667    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773676    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kube-rbac-proxy"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773682    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kube-rbac-proxy"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773693    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kube-rbac-proxy"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773698    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kube-rbac-proxy"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773708    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773712    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773717    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773723    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773731    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6"
Sep 09 21:04:33.773737 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773735    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6"
Sep 09 21:04:33.774148 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773786    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kserve-container"
Sep 09 21:04:33.774148 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773793    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kserve-container"
Sep 09 21:04:33.774148 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773800    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f46f9a-7a7f-4b1f-ad91-8d0035a8c2c0" containerName="kube-rbac-proxy"
Sep 09 21:04:33.774148 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773806    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="768f6010-5ce0-40ad-ab30-a88b9d25b1aa" containerName="switch-graph-6a9b1"
Sep 09 21:04:33.774148 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773812    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3442c5bd-a860-42e5-894f-d5ef7edf7da8" containerName="kube-rbac-proxy"
Sep 09 21:04:33.774148 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.773819    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ba32cd-ef95-459a-81b8-e985dd913c8f" containerName="splitter-graph-7dad6"
Sep 09 21:04:33.778047 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.778027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:33.780904 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.780881    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cmgmq\"/\"openshift-service-ca.crt\""
Sep 09 21:04:33.780904 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.780896    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cmgmq\"/\"kube-root-ca.crt\""
Sep 09 21:04:33.781081 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.780889    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cmgmq\"/\"default-dockercfg-fpm2c\""
Sep 09 21:04:33.786397 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.786374    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cmgmq/must-gather-k52mz"]
Sep 09 21:04:33.840584 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.840548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rt4q\" (UniqueName: \"kubernetes.io/projected/f04d4b1f-75ef-40aa-b501-d1adf6372242-kube-api-access-8rt4q\") pod \"must-gather-k52mz\" (UID: \"f04d4b1f-75ef-40aa-b501-d1adf6372242\") " pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:33.840584 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.840587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f04d4b1f-75ef-40aa-b501-d1adf6372242-must-gather-output\") pod \"must-gather-k52mz\" (UID: \"f04d4b1f-75ef-40aa-b501-d1adf6372242\") " pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:33.941724 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.941688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rt4q\" (UniqueName: \"kubernetes.io/projected/f04d4b1f-75ef-40aa-b501-d1adf6372242-kube-api-access-8rt4q\") pod \"must-gather-k52mz\" (UID: \"f04d4b1f-75ef-40aa-b501-d1adf6372242\") " pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:33.941724 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.941731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f04d4b1f-75ef-40aa-b501-d1adf6372242-must-gather-output\") pod \"must-gather-k52mz\" (UID: \"f04d4b1f-75ef-40aa-b501-d1adf6372242\") " pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:33.942117 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.942096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f04d4b1f-75ef-40aa-b501-d1adf6372242-must-gather-output\") pod \"must-gather-k52mz\" (UID: \"f04d4b1f-75ef-40aa-b501-d1adf6372242\") " pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:33.954601 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:33.954575    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rt4q\" (UniqueName: \"kubernetes.io/projected/f04d4b1f-75ef-40aa-b501-d1adf6372242-kube-api-access-8rt4q\") pod \"must-gather-k52mz\" (UID: \"f04d4b1f-75ef-40aa-b501-d1adf6372242\") " pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:34.087469 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:34.087380    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cmgmq/must-gather-k52mz"
Sep 09 21:04:34.362091 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:34.361988    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cmgmq/must-gather-k52mz"]
Sep 09 21:04:34.365041 ip-10-0-137-234 kubenswrapper[2562]: W0909 21:04:34.364993    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf04d4b1f_75ef_40aa_b501_d1adf6372242.slice/crio-789a7057b37d893b3341060ccf672417b9045ce8b0bc42b7bdaf4dc7150fa32e WatchSource:0}: Error finding container 789a7057b37d893b3341060ccf672417b9045ce8b0bc42b7bdaf4dc7150fa32e: Status 404 returned error can't find the container with id 789a7057b37d893b3341060ccf672417b9045ce8b0bc42b7bdaf4dc7150fa32e
Sep 09 21:04:34.366892 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:34.366875    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:04:34.945075 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:34.945040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cmgmq/must-gather-k52mz" event={"ID":"f04d4b1f-75ef-40aa-b501-d1adf6372242","Type":"ContainerStarted","Data":"789a7057b37d893b3341060ccf672417b9045ce8b0bc42b7bdaf4dc7150fa32e"}
Sep 09 21:04:35.951634 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:35.951597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cmgmq/must-gather-k52mz" event={"ID":"f04d4b1f-75ef-40aa-b501-d1adf6372242","Type":"ContainerStarted","Data":"6ae31f5cde67b723a2db7b439b373b70396ffa90315041ed68f7d9fb804d1b70"}
Sep 09 21:04:35.952036 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:35.951647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cmgmq/must-gather-k52mz" event={"ID":"f04d4b1f-75ef-40aa-b501-d1adf6372242","Type":"ContainerStarted","Data":"64683b49189503060b7523a62b6e610a9f035af90974148ef7f5e9ab46a2ce0e"}
Sep 09 21:04:35.970488 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:35.970427    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cmgmq/must-gather-k52mz" podStartSLOduration=2.132856409 podStartE2EDuration="2.970404866s" totalImagesPullingTime="837.548457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:35.968939001 +0000 UTC m=+3191.942709065" watchObservedRunningTime="2026-09-09 21:04:35.970404866 +0000 UTC m=+3191.944174946"
Sep 09 21:04:36.683541 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:36.683511    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qkfk2_a80d1370-4cbe-4f65-8dd5-d42457f347b2/global-pull-secret-syncer/0.log"
Sep 09 21:04:36.818149 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:36.818110    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lkdsq_1433c535-6ce0-4f9b-b94c-a42e534ff3ee/konnectivity-agent/0.log"
Sep 09 21:04:36.910121 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:36.910077    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-234.ec2.internal_2f674f484fa389b34823ea538de3d17f/haproxy/0.log"
Sep 09 21:04:41.136833 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:41.136796    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-lhznr_f9482af9-54ad-4ac9-8b2d-3c1ad7c16ee0/cluster-monitoring-operator/0.log"
Sep 09 21:04:41.395903 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:41.395814    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2c49z_a8859db1-e24a-4ce2-9962-4f7c2ae74ca0/node-exporter/0.log"
Sep 09 21:04:41.445463 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:41.445433    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2c49z_a8859db1-e24a-4ce2-9962-4f7c2ae74ca0/kube-rbac-proxy/0.log"
Sep 09 21:04:41.487151 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:41.487122    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2c49z_a8859db1-e24a-4ce2-9962-4f7c2ae74ca0/init-textfile/0.log"
Sep 09 21:04:41.876275 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:41.876218    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n2dfq_c02d415d-ee36-4b41-abfc-9d7145ac0895/kube-rbac-proxy-main/0.log"
Sep 09 21:04:41.900971 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:41.900933    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n2dfq_c02d415d-ee36-4b41-abfc-9d7145ac0895/kube-rbac-proxy-self/0.log"
Sep 09 21:04:41.934320 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:41.934275    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n2dfq_c02d415d-ee36-4b41-abfc-9d7145ac0895/openshift-state-metrics/0.log"
Sep 09 21:04:42.214449 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:42.214365    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-hx8mr_bda2d0d5-3962-4ae8-833d-e025a5fc532b/prometheus-operator/0.log"
Sep 09 21:04:42.243150 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:42.243120    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-hx8mr_bda2d0d5-3962-4ae8-833d-e025a5fc532b/kube-rbac-proxy/0.log"
Sep 09 21:04:42.280080 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:42.280046    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-rx2zk_58487adc-4fd6-46c1-871c-974bad59f921/prometheus-operator-admission-webhook/0.log"
Sep 09 21:04:44.206816 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:44.206779    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/1.log"
Sep 09 21:04:44.215667 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:44.215633    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-k99c5_b3ceae69-6511-4bd5-ba0e-c4a2fbbbb38a/console-operator/2.log"
Sep 09 21:04:44.620568 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:44.620487    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75c8b7c86-dpgnm_5ff147f7-a7f1-4ea2-a9ea-237f78482196/console/0.log"
Sep 09 21:04:44.938623 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:44.938580    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"]
Sep 09 21:04:44.943883 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:44.943854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:44.950853 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:44.950794    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"]
Sep 09 21:04:45.046165 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.046127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-podres\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.046327 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.046181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcwl\" (UniqueName: \"kubernetes.io/projected/c7d65def-a0bc-4e53-a02e-49a88652689f-kube-api-access-qmcwl\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.046327 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.046277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-proc\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.046422 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.046340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-sys\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.046422 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.046390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-lib-modules\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147720 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-sys\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147720 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-lib-modules\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147975 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-podres\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147975 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147831    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-sys\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147975 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcwl\" (UniqueName: \"kubernetes.io/projected/c7d65def-a0bc-4e53-a02e-49a88652689f-kube-api-access-qmcwl\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147975 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-podres\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147975 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-lib-modules\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.147975 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-proc\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.148338 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.147987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7d65def-a0bc-4e53-a02e-49a88652689f-proc\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.156665 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.156640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcwl\" (UniqueName: \"kubernetes.io/projected/c7d65def-a0bc-4e53-a02e-49a88652689f-kube-api-access-qmcwl\") pod \"perf-node-gather-daemonset-glctd\" (UID: \"c7d65def-a0bc-4e53-a02e-49a88652689f\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.221658 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.221583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-rj5j7_e8ee55b2-06d4-4686-b4f4-a2e52559a2ee/volume-data-source-validator/0.log"
Sep 09 21:04:45.258567 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.258533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:45.567562 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:45.567530    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"]
Sep 09 21:04:45.570689 ip-10-0-137-234 kubenswrapper[2562]: W0909 21:04:45.570526    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc7d65def_a0bc_4e53_a02e_49a88652689f.slice/crio-a08582bed71f72a1db13193287fdf2d00b8357d932dc8233d56b42dd188a5b8a WatchSource:0}: Error finding container a08582bed71f72a1db13193287fdf2d00b8357d932dc8233d56b42dd188a5b8a: Status 404 returned error can't find the container with id a08582bed71f72a1db13193287fdf2d00b8357d932dc8233d56b42dd188a5b8a
Sep 09 21:04:46.012996 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.012953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd" event={"ID":"c7d65def-a0bc-4e53-a02e-49a88652689f","Type":"ContainerStarted","Data":"02e301ace205374beca5b90d154dfc2a248fc56c511ea28132e1ada038768ea3"}
Sep 09 21:04:46.013257 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.013047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd" event={"ID":"c7d65def-a0bc-4e53-a02e-49a88652689f","Type":"ContainerStarted","Data":"a08582bed71f72a1db13193287fdf2d00b8357d932dc8233d56b42dd188a5b8a"}
Sep 09 21:04:46.013896 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.013861    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:46.034098 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.034040    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd" podStartSLOduration=2.034024301 podStartE2EDuration="2.034024301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:46.031615974 +0000 UTC m=+3202.005386038" watchObservedRunningTime="2026-09-09 21:04:46.034024301 +0000 UTC m=+3202.007794361"
Sep 09 21:04:46.187850 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.187817    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jghvp_a6313d4f-b898-4da8-8010-c67a8726dfea/dns/0.log"
Sep 09 21:04:46.211837 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.211808    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jghvp_a6313d4f-b898-4da8-8010-c67a8726dfea/kube-rbac-proxy/0.log"
Sep 09 21:04:46.292521 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.292435    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2cdv6_cea6223c-9ead-467f-b0e8-c09ceb9ec51a/dns-node-resolver/0.log"
Sep 09 21:04:46.806861 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.806829    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d957f96b-g54nh_d21b8b49-a6b0-4b27-9805-e7030cbd72bd/registry/0.log"
Sep 09 21:04:46.884834 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:46.884805    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6pfl_512ceb34-1d30-47d0-b1f4-e2a0871e519b/node-ca/0.log"
Sep 09 21:04:48.037499 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:48.037469    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s5bqc_66c4d70d-5928-4cc4-82c6-3179c6999aa7/serve-healthcheck-canary/0.log"
Sep 09 21:04:48.406164 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:48.406137    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-xchkg_41ae5d00-e0ab-425b-bb62-7452b59c2101/insights-operator/0.log"
Sep 09 21:04:48.406516 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:48.406493    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-xchkg_41ae5d00-e0ab-425b-bb62-7452b59c2101/insights-operator/1.log"
Sep 09 21:04:48.573364 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:48.573333    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pdkmv_51288a2a-3ab9-4f14-9930-a0105a768158/kube-rbac-proxy/0.log"
Sep 09 21:04:48.594165 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:48.594144    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pdkmv_51288a2a-3ab9-4f14-9930-a0105a768158/exporter/0.log"
Sep 09 21:04:48.616948 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:48.616918    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pdkmv_51288a2a-3ab9-4f14-9930-a0105a768158/extractor/0.log"
Sep 09 21:04:50.930722 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:50.930692    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-98pmb_93854588-f364-425a-9a4d-c7b41809582d/server/0.log"
Sep 09 21:04:53.032406 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:53.032376    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-glctd"
Sep 09 21:04:54.928978 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:54.928947    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-k6d79_48c041b3-39ae-40e7-b31a-dde9da515a76/migrator/0.log"
Sep 09 21:04:54.952525 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:54.952490    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-k6d79_48c041b3-39ae-40e7-b31a-dde9da515a76/graceful-termination/0.log"
Sep 09 21:04:56.313731 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.313694    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bm5wh_c038dca1-e9c9-4c1a-aefb-4125c06f7371/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:04:56.333673 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.333643    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bm5wh_c038dca1-e9c9-4c1a-aefb-4125c06f7371/egress-router-binary-copy/0.log"
Sep 09 21:04:56.353116 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.353084    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bm5wh_c038dca1-e9c9-4c1a-aefb-4125c06f7371/cni-plugins/0.log"
Sep 09 21:04:56.372222 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.372201    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bm5wh_c038dca1-e9c9-4c1a-aefb-4125c06f7371/bond-cni-plugin/0.log"
Sep 09 21:04:56.394309 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.394288    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bm5wh_c038dca1-e9c9-4c1a-aefb-4125c06f7371/routeoverride-cni/0.log"
Sep 09 21:04:56.418097 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.418068    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bm5wh_c038dca1-e9c9-4c1a-aefb-4125c06f7371/whereabouts-cni-bincopy/0.log"
Sep 09 21:04:56.441445 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.441420    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bm5wh_c038dca1-e9c9-4c1a-aefb-4125c06f7371/whereabouts-cni/0.log"
Sep 09 21:04:56.858052 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:56.858026    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k5bpz_b0762481-ab10-46dd-a065-727d330ce5e0/kube-multus/0.log"
Sep 09 21:04:57.042429 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:57.042363    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zq8t9_b355c14c-96e4-4d5d-ada5-6142cd99e951/network-metrics-daemon/0.log"
Sep 09 21:04:57.060244 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:57.060218    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zq8t9_b355c14c-96e4-4d5d-ada5-6142cd99e951/kube-rbac-proxy/0.log"
Sep 09 21:04:58.214131 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.214078    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/ovn-controller/0.log"
Sep 09 21:04:58.232035 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.231991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/ovn-acl-logging/0.log"
Sep 09 21:04:58.263731 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.263707    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/kube-rbac-proxy-node/0.log"
Sep 09 21:04:58.289638 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.289617    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 21:04:58.311293 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.311250    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/northd/0.log"
Sep 09 21:04:58.332269 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.332242    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/nbdb/0.log"
Sep 09 21:04:58.351840 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.351821    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/sbdb/0.log"
Sep 09 21:04:58.411608 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:58.411571    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xt4pz_84e0d8c1-8676-409f-96e7-ff26d02cd1f7/ovnkube-controller/0.log"
Sep 09 21:04:59.379764 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:59.379728    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-5h772_1c9804fd-bdb0-47b7-a668-efdfab71a1d0/check-endpoints/0.log"
Sep 09 21:04:59.463658 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:04:59.463625    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sdx6r_f379b9b0-301a-47af-8703-b07da27e797e/network-check-target-container/0.log"
Sep 09 21:05:00.488363 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:05:00.488332    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g24n5_6e1cd48d-d40b-4a81-beb9-5a62463c96ca/iptables-alerter/0.log"
Sep 09 21:05:01.193636 ip-10-0-137-234 kubenswrapper[2562]: I0909 21:05:01.193607    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7dzkh_8335a8ba-a535-4f1b-b63d-285c7107f06c/tuned/0.log"