Sep 16 12:47:53.429093 ip-10-0-139-96 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:47:53.851132 ip-10-0-139-96 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:53.851132 ip-10-0-139-96 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:47:53.851132 ip-10-0-139-96 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:53.851132 ip-10-0-139-96 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:47:53.851132 ip-10-0-139-96 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:53.852236 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.852157    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:47:53.856042 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856028    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:53.856042 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856043    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856046    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856051    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856056    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856059    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856062    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856065    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856069    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856071    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856074    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856077    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856080    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856082    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856085    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856088    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856090    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856093    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856097    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856100    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:53.856101 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856102    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856105    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856108    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856111    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856114    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856116    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856119    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856121    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856124    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856126    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856129    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856132    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856134    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856137    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856140    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856142    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856144    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856147    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856150    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856154    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:53.856551 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856157    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856159    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856162    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856164    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856168    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856170    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856172    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856175    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856177    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856179    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856182    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856185    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856187    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856191    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856194    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856196    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856199    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856201    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856204    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856207    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:53.857376 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856209    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856212    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856214    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856216    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856219    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856222    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856224    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856227    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856229    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856232    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856235    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856238    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856240    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856243    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856245    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856248    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856250    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856254    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856257    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856260    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:53.857895 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856262    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856265    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856268    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856270    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856277    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856279    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856284    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856287    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856290    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856293    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856295    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856298    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856300    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856303    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856305    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856308    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856310    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856313    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.856315    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:53.858375 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858238    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858248    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858252    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858256    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858260    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858264    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858268    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858271    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858274    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858278    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858280    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858283    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858285    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858288    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858290    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858293    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858295    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858298    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858301    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:53.858831 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858304    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858307    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858309    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858312    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858315    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858319    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858322    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858325    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858328    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858331    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858333    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858336    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858339    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858341    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858345    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858348    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858350    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858353    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858355    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:53.859278 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858358    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858360    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858363    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858365    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858367    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858371    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858373    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858376    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858378    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858381    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858385    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858388    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858392    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858394    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858397    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858399    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858402    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858404    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858407    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858409    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:53.859741 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858411    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858414    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858417    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858419    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858421    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858424    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858426    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858430    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858433    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858436    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858438    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858440    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858443    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858446    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858448    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858450    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858453    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858455    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858458    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858461    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:53.860234 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858463    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858466    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858469    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858471    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858474    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858477    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858479    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858482    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858484    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858487    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858489    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858492    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858495    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858497    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858499    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858502    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858504    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858507    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858509    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858513    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:53.860706 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858516    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858880    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858885    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858888    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858891    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858894    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858896    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858899    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858901    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858904    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858906    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858910    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858914    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858917    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858919    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858922    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858924    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858927    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858929    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858932    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:53.861214 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858935    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858937    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858939    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858942    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858944    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858947    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858949    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858953    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858956    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858959    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858962    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858965    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858968    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858970    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858973    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858976    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858978    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858981    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858983    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:53.861702 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858986    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858989    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858991    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858994    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858996    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.858999    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859001    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859004    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859006    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859008    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859011    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859014    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859016    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859018    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859021    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859023    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859026    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859029    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859031    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859034    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:53.862169 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859036    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859039    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859041    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859044    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859046    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859049    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859051    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859054    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859057    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859059    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859062    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859064    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859066    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859070    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859072    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859075    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859077    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859080    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859082    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:53.862668 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859085    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859088    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859091    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859094    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859097    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859100    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859103    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859106    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859109    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859112    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859114    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859117    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859119    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859122    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859124    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859127    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859129    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859132    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859134    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:53.863126 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859137    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859139    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859142    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859211    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859253    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859260    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859264    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859268    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859272    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859276    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859280    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859284    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859287    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859290    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859293    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859296    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859299    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859302    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859305    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859310    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859313    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859316    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859320    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859323    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:47:53.863613 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859326    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859329    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859337    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859340    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859343    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859347    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859352    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859355    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859358    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859361    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859363    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859367    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859370    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859373    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859376    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859379    2558 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859382    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859385    2558 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859388    2558 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859391    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859394    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859397    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859402    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859405    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859408    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:47:53.864220 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859411    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859413    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859416    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859420    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859422    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859426    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859429    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859431    2558 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859435    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859438    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859441    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859444    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859447    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859451    2558 flags.go:64] FLAG: --help="false"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859454    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-139-96.ec2.internal"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859457    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859460    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859463    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859466    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859469    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859472    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859475    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859477    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:47:53.864798 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859480    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859483    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859486    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859489    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859492    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859494    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859497    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859500    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859503    2558 flags.go:64] FLAG: --lock-file=""
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859506    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859508    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859511    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859517    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859520    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859523    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859525    2558 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859528    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859531    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859534    2558 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859537    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859541    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859544    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859549    2558 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859552    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859555    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:47:53.865411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859557    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859560    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859563    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859566    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859569    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859576    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859578    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859581    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859584    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859587    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859593    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859596    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859599    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859602    2558 flags.go:64] FLAG: --port="10250"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859605    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859608    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0405a4d0bdcffd083"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859611    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859614    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859617    2558 flags.go:64] FLAG: --register-node="true"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859620    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859628    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859631    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859634    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859637    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859641    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:47:53.866026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859644    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859647    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859650    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859652    2558 flags.go:64] FLAG: --runonce="false"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859657    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859661    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859664    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859667    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859670    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859673    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859676    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859679    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859682    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859685    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859687    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859690    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859693    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859696    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859699    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859704    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859707    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859709    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859714    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859716    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859719    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:47:53.866618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859722    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859725    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859728    2558 flags.go:64] FLAG: --v="2"
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859731    2558 flags.go:64] FLAG: --version="false"
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859736    2558 flags.go:64] FLAG: --vmodule=""
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859740    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.859743    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859882    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859886    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859890    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859893    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859898    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859902    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859905    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859908    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859911    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859914    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859916    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859919    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859921    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859924    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:53.867239 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859926    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859929    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859931    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859934    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859937    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859939    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859942    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859944    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859947    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859949    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859952    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859955    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859957    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859960    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859962    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859965    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859968    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859970    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859973    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859976    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:53.867782 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859978    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859981    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859985    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859989    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859992    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859995    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.859997    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860001    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860005    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860008    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860010    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860013    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860016    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860019    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860021    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860024    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860026    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860029    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860031    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:53.868268 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860034    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860036    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860039    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860041    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860044    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860047    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860051    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860054    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860057    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860060    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860064    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860066    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860069    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860072    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860075    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860079    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860083    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860086    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860089    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:53.868731 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860091    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860094    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860097    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860100    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860102    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860105    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860107    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860110    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860112    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860115    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860118    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860120    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860123    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860125    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860128    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860130    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860132    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860135    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860137    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860140    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:53.869185 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860142    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860145    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860147    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860150    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860153    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860156    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.860158    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.861001    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.868401    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.868416    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.869298    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.869305    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869366    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869371    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:53.869676 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869375    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869378    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869381    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869384    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869387    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869390    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869393    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869396    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869398    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869401    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869404    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869406    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869409    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869412    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869414    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869417    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869421    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869424    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869427    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869430    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:53.870095 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869433    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869436    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869438    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869441    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869444    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869447    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869450    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869454    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869458    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869461    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869463    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869466    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869468    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869471    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869473    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869476    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869478    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869481    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869483    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:53.870561 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869486    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869489    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869491    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869493    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869496    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869499    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869502    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869504    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869507    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869509    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869512    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869514    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869517    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869520    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869522    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869525    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869527    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869530    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869533    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869536    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:53.871056 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869540    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869543    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869546    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869549    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869551    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869553    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869556    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869558    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869561    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869563    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869566    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869568    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869571    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869573    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869576    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869578    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869581    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869583    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869586    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:53.871523 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869588    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869591    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869594    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869596    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869599    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869601    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869603    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869606    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869608    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869611    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869614    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869616    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869618    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869621    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869624    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869627    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869630    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869632    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:53.871992 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869634    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.869640    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869742    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869747    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869751    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869754    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869757    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869759    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869762    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869765    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869767    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869770    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869773    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869776    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869778    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:53.872424 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869781    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869783    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869786    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869788    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869791    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869794    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869797    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869801    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869803    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869806    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869824    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869829    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869832    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869835    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869838    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869842    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869844    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869847    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869850    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:53.872795 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869853    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869855    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869858    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869860    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869863    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869865    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869867    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869870    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869872    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869875    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869877    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869880    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869883    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869885    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869888    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869891    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869894    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869896    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869899    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869901    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:53.873256 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869904    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869906    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869909    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869911    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869914    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869916    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869919    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869922    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869924    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869928    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869930    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869933    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869935    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869938    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869940    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869943    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869945    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869948    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869950    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869953    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:53.873746 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869956    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869958    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869961    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869963    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869966    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869968    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869971    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869973    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869976    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869978    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869981    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869983    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869986    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869988    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869991    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869994    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.869998    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870001    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870005    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:53.874269 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870008    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870011    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870014    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870017    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870020    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870023    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870025    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:53.870028    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.870033    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.870123    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.872394    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.873441    2558 server.go:1019] "Starting client certificate rotation"
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.873539    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:53.874718 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.874201    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:53.876354 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.876341    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:53.877406 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.877388    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:53.886144 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.886124    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:47:53.889226 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.889208    2558 log.go:25] "Validated CRI v1 image API"
Sep 16 12:47:53.890066 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.890051    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:47:53.895050 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.895029    2558 fs.go:135] Filesystem UUIDs: map[00bdfd85-0238-419f-9634-3ad53be60c46:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a00a984a-afc3-4c32-85bd-d35775a743d1:/dev/nvme0n1p4]
Sep 16 12:47:53.895131 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.895049    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:47:53.900853 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.900729    2558 manager.go:216] Machine: {Timestamp:2026-09-16 12:47:53.899434502 +0000 UTC m=+0.361063188 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3202134 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b730b9f4ac6e11d484bdce956aeb SystemUUID:ec25b730-b9f4-ac6e-11d4-84bdce956aeb BootID:bfabcff3-b1a2-417c-bbe7-582ad93cc819 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:c3:e0:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:c3:e0:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:9e:d6:2c:db:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:47:53.900853 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.900846    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:47:53.901002 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.900942    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:47:53.904646 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.904618    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:47:53.904840 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.904652    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:47:53.904971 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.904955    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:47:53.905517 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.905505    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:47:53.905569 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.905523    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:47:53.905569 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.905534    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:53.905569 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.905554    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:53.906924 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.906910    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:47:53.911498 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.911484    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:53.911623 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.911613    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:47:53.911687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.911650    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:47:53.911687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.911665    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:47:53.911687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.911678    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:47:53.911805 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.911693    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:47:53.911805 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.911704    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:47:53.912067 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.912054    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:53.912123 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.912076    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:53.917893 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.917878    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:47:53.920130 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.920115    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:47:53.920191 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.920145    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:47:53.921860 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:53.921840    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:47:53.921913 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:53.921840    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:47:53.922486 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922466    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922491    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922510    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922516    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922524    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922529    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922543    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922551    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:47:53.922555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922557    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:47:53.922806 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922562    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:47:53.922806 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922579    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:47:53.922806 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.922584    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:47:53.924350 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.924340    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:47:53.924350 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.924349    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:47:53.929646 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.929634    2558 server.go:1268] "Started kubelet"
Sep 16 12:47:53.929779 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.929724    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:47:53.929875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.929751    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:47:53.929875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.929806    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:47:53.934479 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.934328    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:47:53.934616 ip-10-0-139-96 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:47:53.938700 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.938686    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:47:53.940010 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:53.939982    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:47:53.942096 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.942079    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:47:53.943306 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:53.942125    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-96.ec2.internal.18d5cdf3cbcd21b5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-96.ec2.internal,UID:ip-10-0-139-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-96.ec2.internal,},FirstTimestamp:2026-09-16 12:47:53.929605557 +0000 UTC m=+0.391234244,LastTimestamp:2026-09-16 12:47:53.929605557 +0000 UTC m=+0.391234244,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-96.ec2.internal,}"
Sep 16 12:47:53.945505 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.945486    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:47:53.945505 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.945492    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:47:53.945655 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:53.945567    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:53.946284 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946262    2558 factory.go:55] Registering systemd factory
Sep 16 12:47:53.946284 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946286    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:47:53.946395 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946293    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:47:53.946395 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946306    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:47:53.946395 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946269    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:47:53.946539 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946442    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:47:53.946539 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946452    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:47:53.946539 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946470    2558 factory.go:153] Registering CRI-O factory
Sep 16 12:47:53.946539 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946484    2558 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:47:53.946706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946550    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:47:53.946706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946574    2558 factory.go:103] Registering Raw factory
Sep 16 12:47:53.946706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.946591    2558 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:47:53.947045 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.947033    2558 manager.go:349] Starting recovery of all containers
Sep 16 12:47:53.947486 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:53.947463    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:47:53.947566 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:53.947550    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:47:53.956853 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.956828    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hddfd"
Sep 16 12:47:53.958272 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.958258    2558 manager.go:354] Recovery completed
Sep 16 12:47:53.962389 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.962378    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:53.964432 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.964401    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hddfd"
Sep 16 12:47:53.964970 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.964955    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:53.965026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.964979    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:53.965026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.964997    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:53.966207 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.966188    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:47:53.966207 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.966207    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:47:53.966321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.966226    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:53.966321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.966255    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:53.967143 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.967123    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:53.967215 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.967167    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:53.967215 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.967183    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:53.968456 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.968436    2558 policy_none.go:49] "None policy: Start"
Sep 16 12:47:53.968456 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.968456    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:47:53.968555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.968466    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:47:53.970663 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:53.970650    2558 policy_none.go:47] "Start"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.003894    2558 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.003923    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.003933    2558 server.go:87] "Starting device plugin registration server"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.004200    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.004261    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.004327    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.004406    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.004415    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.004885    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.004925    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.035054    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.036054    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.036091    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.036112    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.036147    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.037968    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.104732    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.107421    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.107447    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.107457    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:54.111321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.107481    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.116151 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.116137    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.116190 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.116156    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-96.ec2.internal\": node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.136899 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.136875    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal"]
Sep 16 12:47:54.136968 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.136928    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:54.138168 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.138154    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:54.138240 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.138180    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:54.138240 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.138193    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:54.140541 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.140529    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:54.140668 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.140645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.140700 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.140684    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:54.141183 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.141165    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:54.141281 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.141191    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:54.141281 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.141201    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:54.141281 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.141256    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:54.141281 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.141273    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:54.141281 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.141281    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:54.143535 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.143520    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.143604 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.143552    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:54.144260 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.144245    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:54.144352 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.144268    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:54.144352 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.144277    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:54.144480 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.144462    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.146828 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.146799    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bda2fdded52a5ad008801928494ecc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal\" (UID: \"f7bda2fdded52a5ad008801928494ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.146908 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.146844    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b846e37f4ef604879d01c53444ca7848-config\") pod \"kube-apiserver-proxy-ip-10-0-139-96.ec2.internal\" (UID: \"b846e37f4ef604879d01c53444ca7848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.146908 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.146869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7bda2fdded52a5ad008801928494ecc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal\" (UID: \"f7bda2fdded52a5ad008801928494ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.174381 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.174360    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-96.ec2.internal\" not found" node="ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.178667 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.178652    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-96.ec2.internal\" not found" node="ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.245517 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.245493    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.247912 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.247896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7bda2fdded52a5ad008801928494ecc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal\" (UID: \"f7bda2fdded52a5ad008801928494ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.248018 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.247921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bda2fdded52a5ad008801928494ecc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal\" (UID: \"f7bda2fdded52a5ad008801928494ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.248018 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.247938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b846e37f4ef604879d01c53444ca7848-config\") pod \"kube-apiserver-proxy-ip-10-0-139-96.ec2.internal\" (UID: \"b846e37f4ef604879d01c53444ca7848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.248018 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.248005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7bda2fdded52a5ad008801928494ecc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal\" (UID: \"f7bda2fdded52a5ad008801928494ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.248112 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.248006    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bda2fdded52a5ad008801928494ecc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal\" (UID: \"f7bda2fdded52a5ad008801928494ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.248112 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.248009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b846e37f4ef604879d01c53444ca7848-config\") pod \"kube-apiserver-proxy-ip-10-0-139-96.ec2.internal\" (UID: \"b846e37f4ef604879d01c53444ca7848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.346055 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.346031    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.446619 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.446553    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.476875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.476854    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.481671 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.481596    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:54.547407 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.547364    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.647837 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.647790    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.749703 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.749629    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.850384 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.850351    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.872930 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.872908    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:47:54.873334 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.873051    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:47:54.937884 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.937866    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:54.950950 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:54.950924    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:54.970960 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.970924    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:42:54 +0000 UTC" deadline="2028-06-05 17:13:50.725506582 +0000 UTC"
Sep 16 12:47:54.970960 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:54.970956    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15076h25m55.754555353s"
Sep 16 12:47:55.048931 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.048865    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:55.051059 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.051042    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:55.074654 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:55.074629    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb846e37f4ef604879d01c53444ca7848.slice/crio-3c07f811e7aa95fda330cdf54d555ab8ed6539d8dafcf12d990d735220f45684 WatchSource:0}: Error finding container 3c07f811e7aa95fda330cdf54d555ab8ed6539d8dafcf12d990d735220f45684: Status 404 returned error can't find the container with id 3c07f811e7aa95fda330cdf54d555ab8ed6539d8dafcf12d990d735220f45684
Sep 16 12:47:55.079815 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.079799    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:47:55.151947 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.151921    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:55.253101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.253073    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:55.353850 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.353762    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:55.454399 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.454369    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-96.ec2.internal\" not found"
Sep 16 12:47:55.470752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.470726    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:55.546587 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.546345    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:55.561939 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.561915    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:47:55.563141 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.563103    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal"
Sep 16 12:47:55.579485 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.579463    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:47:55.903766 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.903733    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:55.913111 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.913085    2558 apiserver.go:52] "Watching apiserver"
Sep 16 12:47:55.923702 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.923338    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:47:55.925436 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.925407    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-njp7q","kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal","openshift-image-registry/node-ca-mqv9c","openshift-multus/multus-additional-cni-plugins-tt472","openshift-multus/multus-nn9ht","openshift-network-diagnostics/network-check-target-qs6rn","openshift-network-operator/iptables-alerter-7qrml","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz","openshift-cluster-node-tuning-operator/tuned-f2f9h","openshift-dns/node-resolver-d2szc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal","openshift-multus/network-metrics-daemon-cblhw","openshift-ovn-kubernetes/ovnkube-node-wb288","kube-system/global-pull-secret-syncer-2mddz"]
Sep 16 12:47:55.928171 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.928147    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:55.931735 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.931171    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:47:55.931735 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.931247    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:47:55.931735 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.931428    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:47:55.931735 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.931465    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b49k8\""
Sep 16 12:47:55.931735 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.931514    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.934267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.933903    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:55.934554 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.934535    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tw7g\""
Sep 16 12:47:55.934755 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.934740    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:47:55.935053 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.934915    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:47:55.935053 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.934968    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:47:55.935322 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.935293    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:47:55.935423 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.935370    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:47:55.935485 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.935468    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:47:55.936591 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.936567    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:47:55.937789 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.936895    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlh26\""
Sep 16 12:47:55.937789 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.936944    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.937789 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.937132    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:47:55.937789 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.937490    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:47:55.939339 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.939309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.939990 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.939781    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rj655\""
Sep 16 12:47:55.939990 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.939851    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:47:55.940139 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.940102    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:47:55.940139 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.940120    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:47:55.940241 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.940140    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:47:55.940241 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.940215    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:47:55.941994 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.941725    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:47:55.941994 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.941831    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:47:55.941994 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.941841    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:47:55.942189 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.942165    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6qbgp\""
Sep 16 12:47:55.944355 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.944336    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.946346 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.946329    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:47:55.947353 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.947062    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:55.947437 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.947416    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vxvw8\""
Sep 16 12:47:55.947513 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.947448    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:47:55.947747 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.947727    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:47:55.949821 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.949774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:55.949902 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.949872    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:47:55.949976 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.949952    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wtwnh\""
Sep 16 12:47:55.950055 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.950033    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:47:55.950460 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.950442    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:47:55.952443 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.952424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:55.955500 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.955110    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:55.955628 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.955587    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:47:55.956657 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.956634    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:47:55.956837 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.956815    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zq9w2\""
Sep 16 12:47:55.959330 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.958942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:55.959330 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:55.959060    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:47:55.959330 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.959188    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:47:55.959550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.959359    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9222l\""
Sep 16 12:47:55.961092 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.959878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-cni-netd\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.961187 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961134    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovn-node-metrics-cert\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.961187 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-cnibin\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.961318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961205    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-conf-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.961318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961233    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysctl-d\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.961318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.960041    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:47:55.961318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c370ba8b-56d2-46e1-8ed2-baae87e9df49-agent-certs\") pod \"konnectivity-agent-njp7q\" (UID: \"c370ba8b-56d2-46e1-8ed2-baae87e9df49\") " pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:55.961318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-etc-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-ovn\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23acb97-072a-4d73-9411-6aee7d0b9259-host\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.960022    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysconfig\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-kubernetes\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961506    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-node-log\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l7q\" (UniqueName: \"kubernetes.io/projected/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-kube-api-access-x5l7q\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.961550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysctl-conf\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961594    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-systemd\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-tuned\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snx7h\" (UniqueName: \"kubernetes.io/projected/93857f18-85a0-4668-87ab-8c83030b026c-kube-api-access-snx7h\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961776    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-k8s-cni-cncf-io\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25dzv\" (UniqueName: \"kubernetes.io/projected/be79be6c-655b-4a79-be76-7fb70374c8c6-kube-api-access-25dzv\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961852    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-daemon-config\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-etc-kubernetes\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.961961    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdxp\" (UniqueName: \"kubernetes.io/projected/e23acb97-072a-4d73-9411-6aee7d0b9259-kube-api-access-dzdxp\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:55.962043 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-tmp\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-env-overrides\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-hostroot\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962202    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-var-lib-kubelet\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-systemd-units\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962297    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-cni-bin\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962337    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovnkube-config\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962377    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-cnibin\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-cni-multus\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962459    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be79be6c-655b-4a79-be76-7fb70374c8c6-host-slash\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:55.962509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c370ba8b-56d2-46e1-8ed2-baae87e9df49-konnectivity-ca\") pod \"konnectivity-agent-njp7q\" (UID: \"c370ba8b-56d2-46e1-8ed2-baae87e9df49\") " pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:55.963098 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962537    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-log-socket\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.963098 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-netns\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.963098 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvcx\" (UniqueName: \"kubernetes.io/projected/13252a94-a763-48d6-b1d8-6a4bb134ec77-kube-api-access-5cvcx\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.963098 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.962644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23acb97-072a-4d73-9411-6aee7d0b9259-serviceca\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:55.963301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963150    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-host\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.963301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963196    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-systemd\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.963301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13252a94-a763-48d6-b1d8-6a4bb134ec77-cni-binary-copy\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.963446 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963291    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-multus-certs\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.963446 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963337    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:47:55.963446 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-sys\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.963446 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963409    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.963446 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-system-cni-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.963692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963467    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-socket-dir-parent\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.963692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-cni-bin\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.963692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963520    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-lib-modules\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.963692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-slash\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.963692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.963692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963621    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be79be6c-655b-4a79-be76-7fb70374c8c6-iptables-alerter-script\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:55.963692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963651    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-run\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963692    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-run-netns\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963758    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-var-lib-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-kubelet\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-modprobe-d\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-kubelet\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963942    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-system-cni-dir\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rx4\" (UniqueName: \"kubernetes.io/projected/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-kube-api-access-w5rx4\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.963996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.964067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovnkube-script-lib\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:55.964121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.964124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-cni-binary-copy\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:55.964663 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.964174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-cni-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:55.964722 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.964694    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:47:55.984664 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.984644    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ptqtw"
Sep 16 12:47:55.997093 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:55.997066    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ptqtw"
Sep 16 12:47:56.040536 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.040478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal" event={"ID":"f7bda2fdded52a5ad008801928494ecc","Type":"ContainerStarted","Data":"2f37866f83e3ffef75a55213c35ced21995a9200cc3d5edf1cc7360b16891306"}
Sep 16 12:47:56.041607 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.041578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal" event={"ID":"b846e37f4ef604879d01c53444ca7848","Type":"ContainerStarted","Data":"3c07f811e7aa95fda330cdf54d555ab8ed6539d8dafcf12d990d735220f45684"}
Sep 16 12:47:56.047618 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.047596    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:47:56.064737 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-system-cni-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.064860 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064743    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-socket-dir-parent\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.064860 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-cni-bin\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.064860 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-cni-bin\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.064860 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064852    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-socket-dir-parent\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.065061 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064876    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-system-cni-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.065061 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-lib-modules\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.065061 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.065061 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064960    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-dbus\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.065061 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.064985    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.065061 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-slash\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065061 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-lib-modules\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-slash\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065094    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be79be6c-655b-4a79-be76-7fb70374c8c6-iptables-alerter-script\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065138    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-run\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065185    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-registration-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065224    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxvhv\" (UniqueName: \"kubernetes.io/projected/020b3985-61c1-4945-897f-1e10c5224294-kube-api-access-sxvhv\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-run\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-run-netns\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065250    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065276    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-var-lib-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065300    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-kubelet\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.065325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-run-netns\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-kubelet\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065365    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-var-lib-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-modprobe-d\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/839b45b7-c79f-440d-ac6b-6390e71c58b6-hosts-file\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-kubelet\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-system-cni-dir\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rx4\" (UniqueName: \"kubernetes.io/projected/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-kube-api-access-w5rx4\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-modprobe-d\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065520    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-kubelet\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npfs\" (UniqueName: \"kubernetes.io/projected/839b45b7-c79f-440d-ac6b-6390e71c58b6-kube-api-access-5npfs\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065551    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-system-cni-dir\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-socket-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065603    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovnkube-script-lib\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065621    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-cni-binary-copy\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-cni-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.065752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065654    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-etc-selinux\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065664    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be79be6c-655b-4a79-be76-7fb70374c8c6-iptables-alerter-script\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-cni-netd\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovn-node-metrics-cert\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065770    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-cnibin\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065796    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-conf-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065819    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysctl-d\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065806    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-cni-netd\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-cnibin\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-cni-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c370ba8b-56d2-46e1-8ed2-baae87e9df49-agent-certs\") pod \"konnectivity-agent-njp7q\" (UID: \"c370ba8b-56d2-46e1-8ed2-baae87e9df49\") " pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-conf-dir\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065886    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-etc-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-ovn\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065924    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-etc-openvswitch\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23acb97-072a-4d73-9411-6aee7d0b9259-host\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.066305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.065991    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysconfig\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066013    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-kubernetes\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysctl-d\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-node-log\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l7q\" (UniqueName: \"kubernetes.io/projected/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-kube-api-access-x5l7q\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysctl-conf\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066110    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-systemd\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-tuned\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snx7h\" (UniqueName: \"kubernetes.io/projected/93857f18-85a0-4668-87ab-8c83030b026c-kube-api-access-snx7h\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-k8s-cni-cncf-io\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066235    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25dzv\" (UniqueName: \"kubernetes.io/projected/be79be6c-655b-4a79-be76-7fb70374c8c6-kube-api-access-25dzv\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066242    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovnkube-script-lib\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-node-log\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066260    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-daemon-config\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23acb97-072a-4d73-9411-6aee7d0b9259-host\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066285    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-ovn\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.066877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066320    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-kubernetes\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066322    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysconfig\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066356    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-systemd\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-sysctl-conf\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-cni-binary-copy\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066506    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-k8s-cni-cncf-io\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066520    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066708    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-etc-kubernetes\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066724    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066735    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdxp\" (UniqueName: \"kubernetes.io/projected/e23acb97-072a-4d73-9411-6aee7d0b9259-kube-api-access-dzdxp\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-tmp\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-env-overrides\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066819    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-hostroot\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-var-lib-kubelet\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066916    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-systemd-units\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.067574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066978    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-cni-bin\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.066999    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-etc-kubernetes\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovnkube-config\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-cni-bin\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067001    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13252a94-a763-48d6-b1d8-6a4bb134ec77-multus-daemon-config\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-systemd-units\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067084    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-hostroot\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-cnibin\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-var-lib-kubelet\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-cni-multus\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-var-lib-cni-multus\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067524    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-env-overrides\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93857f18-85a0-4668-87ab-8c83030b026c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067604    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93857f18-85a0-4668-87ab-8c83030b026c-cnibin\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be79be6c-655b-4a79-be76-7fb70374c8c6-host-slash\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c370ba8b-56d2-46e1-8ed2-baae87e9df49-konnectivity-ca\") pod \"konnectivity-agent-njp7q\" (UID: \"c370ba8b-56d2-46e1-8ed2-baae87e9df49\") " pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067707    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m52v9\" (UniqueName: \"kubernetes.io/projected/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-kube-api-access-m52v9\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-log-socket\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.068368 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067753    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-netns\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be79be6c-655b-4a79-be76-7fb70374c8c6-host-slash\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-log-socket\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvcx\" (UniqueName: \"kubernetes.io/projected/13252a94-a763-48d6-b1d8-6a4bb134ec77-kube-api-access-5cvcx\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-netns\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23acb97-072a-4d73-9411-6aee7d0b9259-serviceca\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-host\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-device-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.067984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839b45b7-c79f-440d-ac6b-6390e71c58b6-tmp-dir\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-systemd\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13252a94-a763-48d6-b1d8-6a4bb134ec77-cni-binary-copy\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-multus-certs\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068081    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-sys\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068120    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-sys-fs\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068142    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-kubelet-config\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.069243 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068277    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-run-systemd\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068367    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c370ba8b-56d2-46e1-8ed2-baae87e9df49-konnectivity-ca\") pod \"konnectivity-agent-njp7q\" (UID: \"c370ba8b-56d2-46e1-8ed2-baae87e9df49\") " pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23acb97-072a-4d73-9411-6aee7d0b9259-serviceca\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068458    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-host\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068559    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-sys\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13252a94-a763-48d6-b1d8-6a4bb134ec77-host-run-multus-certs\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.068805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13252a94-a763-48d6-b1d8-6a4bb134ec77-cni-binary-copy\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.070150 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.069258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovnkube-config\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.071223 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.071122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-tmp\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.071223 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.071159    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-etc-tuned\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.071487 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.071469    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-ovn-node-metrics-cert\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.071585 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.071524    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c370ba8b-56d2-46e1-8ed2-baae87e9df49-agent-certs\") pod \"konnectivity-agent-njp7q\" (UID: \"c370ba8b-56d2-46e1-8ed2-baae87e9df49\") " pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:56.076012 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.075989    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:56.076106 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.076019    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:56.076106 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.076033    2558 projected.go:196] Error preparing data for projected volume kube-api-access-6jf8k for pod openshift-network-diagnostics/network-check-target-qs6rn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:56.076233 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.076163    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k podName:3b346991-b5d0-4414-b808-6380a553c4a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:56.576131325 +0000 UTC m=+3.037760006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6jf8k" (UniqueName: "kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k") pod "network-check-target-qs6rn" (UID: "3b346991-b5d0-4414-b808-6380a553c4a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:56.078487 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.078463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvcx\" (UniqueName: \"kubernetes.io/projected/13252a94-a763-48d6-b1d8-6a4bb134ec77-kube-api-access-5cvcx\") pod \"multus-nn9ht\" (UID: \"13252a94-a763-48d6-b1d8-6a4bb134ec77\") " pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.078944 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.078922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rx4\" (UniqueName: \"kubernetes.io/projected/106cce76-3ca1-4e9e-a859-0e0ad9f9e37a-kube-api-access-w5rx4\") pod \"tuned-f2f9h\" (UID: \"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.079038 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.078972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snx7h\" (UniqueName: \"kubernetes.io/projected/93857f18-85a0-4668-87ab-8c83030b026c-kube-api-access-snx7h\") pod \"multus-additional-cni-plugins-tt472\" (UID: \"93857f18-85a0-4668-87ab-8c83030b026c\") " pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.079104 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.079035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdxp\" (UniqueName: \"kubernetes.io/projected/e23acb97-072a-4d73-9411-6aee7d0b9259-kube-api-access-dzdxp\") pod \"node-ca-mqv9c\" (UID: \"e23acb97-072a-4d73-9411-6aee7d0b9259\") " pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:56.079429 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.079409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l7q\" (UniqueName: \"kubernetes.io/projected/551e18e6-7f02-42aa-8b5f-2132d9df8fcd-kube-api-access-x5l7q\") pod \"ovnkube-node-wb288\" (UID: \"551e18e6-7f02-42aa-8b5f-2132d9df8fcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.080011 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.079993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25dzv\" (UniqueName: \"kubernetes.io/projected/be79be6c-655b-4a79-be76-7fb70374c8c6-kube-api-access-25dzv\") pod \"iptables-alerter-7qrml\" (UID: \"be79be6c-655b-4a79-be76-7fb70374c8c6\") " pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:56.168482 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npfs\" (UniqueName: \"kubernetes.io/projected/839b45b7-c79f-440d-ac6b-6390e71c58b6-kube-api-access-5npfs\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.168482 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-socket-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.168482 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-etc-selinux\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m52v9\" (UniqueName: \"kubernetes.io/projected/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-kube-api-access-m52v9\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-device-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839b45b7-c79f-440d-ac6b-6390e71c58b6-tmp-dir\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-sys-fs\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-socket-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168641    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-kubelet-config\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-dbus\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.168716    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:56.168721 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168752    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-registration-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.168785    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:56.66876575 +0000 UTC m=+3.130394431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-registration-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxvhv\" (UniqueName: \"kubernetes.io/projected/020b3985-61c1-4945-897f-1e10c5224294-kube-api-access-sxvhv\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/839b45b7-c79f-440d-ac6b-6390e71c58b6-hosts-file\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/839b45b7-c79f-440d-ac6b-6390e71c58b6-hosts-file\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839b45b7-c79f-440d-ac6b-6390e71c58b6-tmp-dir\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168962    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-kubelet-config\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-etc-selinux\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.168988    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.169055    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.169052    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-device-dir\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.169075    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/020b3985-61c1-4945-897f-1e10c5224294-sys-fs\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.169100    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret podName:8a48d3d0-74d6-4512-8a11-e79ca407a8e4 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:56.669085954 +0000 UTC m=+3.130714639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret") pod "global-pull-secret-syncer-2mddz" (UID: "8a48d3d0-74d6-4512-8a11-e79ca407a8e4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:56.169142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.169102    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-dbus\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.181584 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.181550    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npfs\" (UniqueName: \"kubernetes.io/projected/839b45b7-c79f-440d-ac6b-6390e71c58b6-kube-api-access-5npfs\") pod \"node-resolver-d2szc\" (UID: \"839b45b7-c79f-440d-ac6b-6390e71c58b6\") " pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.182254 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.182234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxvhv\" (UniqueName: \"kubernetes.io/projected/020b3985-61c1-4945-897f-1e10c5224294-kube-api-access-sxvhv\") pod \"aws-ebs-csi-driver-node-fx4bz\" (UID: \"020b3985-61c1-4945-897f-1e10c5224294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.182700 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.182678    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m52v9\" (UniqueName: \"kubernetes.io/projected/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-kube-api-access-m52v9\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:56.242913 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.242884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qrml"
Sep 16 12:47:56.250817 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.250792    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:47:56.265777 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.265756    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mqv9c"
Sep 16 12:47:56.270894 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.270862    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tt472"
Sep 16 12:47:56.278658 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.278639    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nn9ht"
Sep 16 12:47:56.286424 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.286397    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2f9h"
Sep 16 12:47:56.295235 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.295216    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:47:56.303968 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.303947    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d2szc"
Sep 16 12:47:56.308864 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.308850    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz"
Sep 16 12:47:56.673998 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.673964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:56.674181 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.674026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:47:56.674181 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:56.674050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:56.674181 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674148    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:56.674181 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674151    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:56.674181 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674168    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:56.674374 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674189    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:56.674374 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674202    2558 projected.go:196] Error preparing data for projected volume kube-api-access-6jf8k for pod openshift-network-diagnostics/network-check-target-qs6rn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:56.674374 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674212    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret podName:8a48d3d0-74d6-4512-8a11-e79ca407a8e4 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:57.674198096 +0000 UTC m=+4.135826771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret") pod "global-pull-secret-syncer-2mddz" (UID: "8a48d3d0-74d6-4512-8a11-e79ca407a8e4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:56.674374 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674230    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:57.674221672 +0000 UTC m=+4.135850347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:56.674374 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:56.674254    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k podName:3b346991-b5d0-4414-b808-6380a553c4a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:57.674235281 +0000 UTC m=+4.135863972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jf8k" (UniqueName: "kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k") pod "network-check-target-qs6rn" (UID: "3b346991-b5d0-4414-b808-6380a553c4a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:57.001257 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:57.001175    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:42:56 +0000 UTC" deadline="2028-05-01 22:00:23.721242877 +0000 UTC"
Sep 16 12:47:57.001257 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:57.001206    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14241h12m26.720040359s"
Sep 16 12:47:57.106938 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.106898    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93857f18_85a0_4668_87ab_8c83030b026c.slice/crio-539ee330c3ce33199bcbc782054440228f772e592e82d8ed7dbddb8454e1f821 WatchSource:0}: Error finding container 539ee330c3ce33199bcbc782054440228f772e592e82d8ed7dbddb8454e1f821: Status 404 returned error can't find the container with id 539ee330c3ce33199bcbc782054440228f772e592e82d8ed7dbddb8454e1f821
Sep 16 12:47:57.108593 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.108566    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe79be6c_655b_4a79_be76_7fb70374c8c6.slice/crio-4578a22e58c84006378135edb03ef754b768f6b4d4072f0cf6751160f69cf762 WatchSource:0}: Error finding container 4578a22e58c84006378135edb03ef754b768f6b4d4072f0cf6751160f69cf762: Status 404 returned error can't find the container with id 4578a22e58c84006378135edb03ef754b768f6b4d4072f0cf6751160f69cf762
Sep 16 12:47:57.109465 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.109427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23acb97_072a_4d73_9411_6aee7d0b9259.slice/crio-1b8b82ce4b2a56e24d5ac7ce2e5cfddf47099d0a74ec69c222cad2654053c748 WatchSource:0}: Error finding container 1b8b82ce4b2a56e24d5ac7ce2e5cfddf47099d0a74ec69c222cad2654053c748: Status 404 returned error can't find the container with id 1b8b82ce4b2a56e24d5ac7ce2e5cfddf47099d0a74ec69c222cad2654053c748
Sep 16 12:47:57.112877 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.112838    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc370ba8b_56d2_46e1_8ed2_baae87e9df49.slice/crio-66efa7859b0d917fb250ea9a1bb1c132d13f74538ea4e02156c4a739662e3eeb WatchSource:0}: Error finding container 66efa7859b0d917fb250ea9a1bb1c132d13f74538ea4e02156c4a739662e3eeb: Status 404 returned error can't find the container with id 66efa7859b0d917fb250ea9a1bb1c132d13f74538ea4e02156c4a739662e3eeb
Sep 16 12:47:57.114314 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.114092    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106cce76_3ca1_4e9e_a859_0e0ad9f9e37a.slice/crio-188cfae1a58d87bc457f03eb3207f79df327037a67fb9a17a29a7ef8cca04d3d WatchSource:0}: Error finding container 188cfae1a58d87bc457f03eb3207f79df327037a67fb9a17a29a7ef8cca04d3d: Status 404 returned error can't find the container with id 188cfae1a58d87bc457f03eb3207f79df327037a67fb9a17a29a7ef8cca04d3d
Sep 16 12:47:57.115165 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.115054    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020b3985_61c1_4945_897f_1e10c5224294.slice/crio-9401fc91818e513447449eb6045ab3773c9e046e0f239854bd9de1a132706c8d WatchSource:0}: Error finding container 9401fc91818e513447449eb6045ab3773c9e046e0f239854bd9de1a132706c8d: Status 404 returned error can't find the container with id 9401fc91818e513447449eb6045ab3773c9e046e0f239854bd9de1a132706c8d
Sep 16 12:47:57.115970 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.115942    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13252a94_a763_48d6_b1d8_6a4bb134ec77.slice/crio-b9baa0fd83c3a13e56427ac6bf0445f18d33a2ea9411129d7ac6d904a50a6323 WatchSource:0}: Error finding container b9baa0fd83c3a13e56427ac6bf0445f18d33a2ea9411129d7ac6d904a50a6323: Status 404 returned error can't find the container with id b9baa0fd83c3a13e56427ac6bf0445f18d33a2ea9411129d7ac6d904a50a6323
Sep 16 12:47:57.118182 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.117986    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551e18e6_7f02_42aa_8b5f_2132d9df8fcd.slice/crio-860837f3ce0ed203e5b5848544706c89b541a73249842fa99c23a5c2f7cfa9d6 WatchSource:0}: Error finding container 860837f3ce0ed203e5b5848544706c89b541a73249842fa99c23a5c2f7cfa9d6: Status 404 returned error can't find the container with id 860837f3ce0ed203e5b5848544706c89b541a73249842fa99c23a5c2f7cfa9d6
Sep 16 12:47:57.118834 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:47:57.118766    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839b45b7_c79f_440d_ac6b_6390e71c58b6.slice/crio-a31866985dd16d6a01b57599f1be13b0c3d543119c2d2df6d6ea229113a2c2f5 WatchSource:0}: Error finding container a31866985dd16d6a01b57599f1be13b0c3d543119c2d2df6d6ea229113a2c2f5: Status 404 returned error can't find the container with id a31866985dd16d6a01b57599f1be13b0c3d543119c2d2df6d6ea229113a2c2f5
Sep 16 12:47:57.682181 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:57.682100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:57.682181 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:57.682174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:57.682432 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:57.682211    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:47:57.682432 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.682364    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:57.682432 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.682387    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:57.682432 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.682400    2558 projected.go:196] Error preparing data for projected volume kube-api-access-6jf8k for pod openshift-network-diagnostics/network-check-target-qs6rn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:57.682634 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.682461    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k podName:3b346991-b5d0-4414-b808-6380a553c4a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:59.682442191 +0000 UTC m=+6.144070888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jf8k" (UniqueName: "kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k") pod "network-check-target-qs6rn" (UID: "3b346991-b5d0-4414-b808-6380a553c4a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:57.682979 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.682916    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:57.683084 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.682990    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret podName:8a48d3d0-74d6-4512-8a11-e79ca407a8e4 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:59.682967369 +0000 UTC m=+6.144596045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret") pod "global-pull-secret-syncer-2mddz" (UID: "8a48d3d0-74d6-4512-8a11-e79ca407a8e4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:57.683084 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.683052    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:57.683209 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:57.683088    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:59.683074945 +0000 UTC m=+6.144703637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:58.002036 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.001936    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:42:56 +0000 UTC" deadline="2028-05-03 19:44:09.286341399 +0000 UTC"
Sep 16 12:47:58.002036 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.001979    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14286h56m11.28436774s"
Sep 16 12:47:58.039680 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.039649    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:58.039839 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:58.039779    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:47:58.040285 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.040208    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:58.040425 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:58.040315    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:47:58.040425 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.040383    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:47:58.040663 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:58.040452    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:47:58.059012 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.058984    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"860837f3ce0ed203e5b5848544706c89b541a73249842fa99c23a5c2f7cfa9d6"}
Sep 16 12:47:58.065509 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.065458    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nn9ht" event={"ID":"13252a94-a763-48d6-b1d8-6a4bb134ec77","Type":"ContainerStarted","Data":"b9baa0fd83c3a13e56427ac6bf0445f18d33a2ea9411129d7ac6d904a50a6323"}
Sep 16 12:47:58.067528 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.067503    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-njp7q" event={"ID":"c370ba8b-56d2-46e1-8ed2-baae87e9df49","Type":"ContainerStarted","Data":"66efa7859b0d917fb250ea9a1bb1c132d13f74538ea4e02156c4a739662e3eeb"}
Sep 16 12:47:58.072258 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.072234    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerStarted","Data":"539ee330c3ce33199bcbc782054440228f772e592e82d8ed7dbddb8454e1f821"}
Sep 16 12:47:58.087471 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.087411    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7bda2fdded52a5ad008801928494ecc" containerID="c7c8b334ebac650f2149b4f02eeacdd1107255ea044b85ac5ed780ee90ce54ee" exitCode=0
Sep 16 12:47:58.091834 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.091810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal" event={"ID":"f7bda2fdded52a5ad008801928494ecc","Type":"ContainerDied","Data":"c7c8b334ebac650f2149b4f02eeacdd1107255ea044b85ac5ed780ee90ce54ee"}
Sep 16 12:47:58.100851 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.099459    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz" event={"ID":"020b3985-61c1-4945-897f-1e10c5224294","Type":"ContainerStarted","Data":"9401fc91818e513447449eb6045ab3773c9e046e0f239854bd9de1a132706c8d"}
Sep 16 12:47:58.117702 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.117652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2f9h" event={"ID":"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a","Type":"ContainerStarted","Data":"188cfae1a58d87bc457f03eb3207f79df327037a67fb9a17a29a7ef8cca04d3d"}
Sep 16 12:47:58.127580 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.127520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mqv9c" event={"ID":"e23acb97-072a-4d73-9411-6aee7d0b9259","Type":"ContainerStarted","Data":"1b8b82ce4b2a56e24d5ac7ce2e5cfddf47099d0a74ec69c222cad2654053c748"}
Sep 16 12:47:58.131873 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.131813    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qrml" event={"ID":"be79be6c-655b-4a79-be76-7fb70374c8c6","Type":"ContainerStarted","Data":"4578a22e58c84006378135edb03ef754b768f6b4d4072f0cf6751160f69cf762"}
Sep 16 12:47:58.135623 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.134987    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal" event={"ID":"b846e37f4ef604879d01c53444ca7848","Type":"ContainerStarted","Data":"080a5aae1a866f5f8e9b8eb474e3821a7ae302c5b166e0415f3eb67929453dc3"}
Sep 16 12:47:58.150692 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.150474    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-96.ec2.internal" podStartSLOduration=3.150453738 podStartE2EDuration="3.150453738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:58.149873809 +0000 UTC m=+4.611502511" watchObservedRunningTime="2026-09-16 12:47:58.150453738 +0000 UTC m=+4.612082438"
Sep 16 12:47:58.171751 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:58.171716    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d2szc" event={"ID":"839b45b7-c79f-440d-ac6b-6390e71c58b6","Type":"ContainerStarted","Data":"a31866985dd16d6a01b57599f1be13b0c3d543119c2d2df6d6ea229113a2c2f5"}
Sep 16 12:47:59.224497 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:59.224355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal" event={"ID":"f7bda2fdded52a5ad008801928494ecc","Type":"ContainerStarted","Data":"a4223fda291e4ee555b17eb6c5ef0dcb1747b4ce75218480c5bad434eff495e5"}
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:59.698953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:59.699017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:47:59.699053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699191    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699255    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret podName:8a48d3d0-74d6-4512-8a11-e79ca407a8e4 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:03.699234772 +0000 UTC m=+10.160863451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret") pod "global-pull-secret-syncer-2mddz" (UID: "8a48d3d0-74d6-4512-8a11-e79ca407a8e4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699653    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699707    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:03.699690749 +0000 UTC m=+10.161319425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699783    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699797    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699824    2558 projected.go:196] Error preparing data for projected volume kube-api-access-6jf8k for pod openshift-network-diagnostics/network-check-target-qs6rn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:59.700022 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:47:59.699865    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k podName:3b346991-b5d0-4414-b808-6380a553c4a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:03.699852595 +0000 UTC m=+10.161481290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jf8k" (UniqueName: "kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k") pod "network-check-target-qs6rn" (UID: "3b346991-b5d0-4414-b808-6380a553c4a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:00.039016 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:00.038916    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:00.039173 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:00.039042    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:00.039593 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:00.039416    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:00.039593 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:00.039523    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:00.040112 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:00.039845    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:00.040112 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:00.039948    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:02.036775 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:02.036741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:02.037210 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:02.036777    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:02.037210 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:02.036741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:02.037210 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:02.036891    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:02.037210 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:02.036996    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:02.037210 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:02.037099    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:03.733224 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:03.732981    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:03.733672 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:03.733261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:03.733672 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:03.733297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:03.733672 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.733139    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:03.733672 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.733469    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:11.733449771 +0000 UTC m=+18.195078468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:03.734096 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.733949    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:03.734096 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.733977    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:03.734096 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.733990    2558 projected.go:196] Error preparing data for projected volume kube-api-access-6jf8k for pod openshift-network-diagnostics/network-check-target-qs6rn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:03.734096 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.734038    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k podName:3b346991-b5d0-4414-b808-6380a553c4a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:11.734022993 +0000 UTC m=+18.195651689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jf8k" (UniqueName: "kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k") pod "network-check-target-qs6rn" (UID: "3b346991-b5d0-4414-b808-6380a553c4a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:03.734096 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.733408    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:03.734096 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:03.734076    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret podName:8a48d3d0-74d6-4512-8a11-e79ca407a8e4 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:11.734066788 +0000 UTC m=+18.195695466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret") pod "global-pull-secret-syncer-2mddz" (UID: "8a48d3d0-74d6-4512-8a11-e79ca407a8e4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:04.039197 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:04.039090    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:04.039479 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:04.039461    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:04.039572 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:04.039422    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:04.040014 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:04.039819    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:04.040014 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:04.039979    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:04.040180 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:04.039723    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:06.036953 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:06.036920    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:06.037412 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:06.037063    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:06.037493 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:06.037444    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:06.037562 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:06.037540    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:06.037680 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:06.037668    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:06.037765 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:06.037747    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:08.037346 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:08.037304    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:08.037836 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:08.037304    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:08.037836 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:08.037431    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:08.037836 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:08.037482    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:08.037836 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:08.037310    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:08.037836 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:08.037570    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:10.036884 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:10.036836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:10.037417 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:10.036837    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:10.037417 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:10.036977    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:10.037417 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:10.036835    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:10.037417 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:10.037035    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:10.037417 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:10.037159    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:11.792796 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:11.792754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:11.792824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:11.792884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.792931    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.792946    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.792964    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.792971    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.792976    2558 projected.go:196] Error preparing data for projected volume kube-api-access-6jf8k for pod openshift-network-diagnostics/network-check-target-qs6rn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.793009    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret podName:8a48d3d0-74d6-4512-8a11-e79ca407a8e4 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:27.792988509 +0000 UTC m=+34.254617205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret") pod "global-pull-secret-syncer-2mddz" (UID: "8a48d3d0-74d6-4512-8a11-e79ca407a8e4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.793024    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k podName:3b346991-b5d0-4414-b808-6380a553c4a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:27.793015035 +0000 UTC m=+34.254643729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jf8k" (UniqueName: "kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k") pod "network-check-target-qs6rn" (UID: "3b346991-b5d0-4414-b808-6380a553c4a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:11.793238 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:11.793040    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:27.793033269 +0000 UTC m=+34.254661949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:12.036914 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:12.036774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:12.036914 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:12.036796    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:12.036914 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:12.036774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:12.037140 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:12.036929    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:12.037140 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:12.037036    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:12.037140 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:12.037101    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:14.037321 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:14.037293    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:14.037829 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:14.037400    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:14.037829 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:14.037495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:14.037829 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:14.037608    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:14.037829 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:14.037654    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:14.037829 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:14.037721    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:15.260978 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:15.260923    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-njp7q" event={"ID":"c370ba8b-56d2-46e1-8ed2-baae87e9df49","Type":"ContainerStarted","Data":"a8091a80b2e1d52f307bb30d18d140405ab3fb5a21afdd399f3179d9b6e6edc4"}
Sep 16 12:48:15.263406 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:15.263313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2f9h" event={"ID":"106cce76-3ca1-4e9e-a859-0e0ad9f9e37a","Type":"ContainerStarted","Data":"8527ad6a7024eabe4df113d0aafb7c306330ae121f175855cc0957b686222211"}
Sep 16 12:48:15.268602 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:15.268565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"2a55ff12e5a15765a9480ea8ed77385f82295a1f008347f31364df896942a47a"}
Sep 16 12:48:15.273301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:15.273242    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nn9ht" event={"ID":"13252a94-a763-48d6-b1d8-6a4bb134ec77","Type":"ContainerStarted","Data":"8fb19df59e99ba032403a9d1f03f7661cbed14d06109ad2a2f7f6fe99cdc3726"}
Sep 16 12:48:15.283481 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:15.283424    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f2f9h" podStartSLOduration=3.333478617 podStartE2EDuration="21.283404236s" totalImagesPullingTime="17.949925619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:15.282783876 +0000 UTC m=+21.744412572" watchObservedRunningTime="2026-09-16 12:48:15.283404236 +0000 UTC m=+21.745032933"
Sep 16 12:48:15.283690 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:15.283654    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-96.ec2.internal" podStartSLOduration=20.283645449 podStartE2EDuration="20.283645449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:59.242188506 +0000 UTC m=+5.703817202" watchObservedRunningTime="2026-09-16 12:48:15.283645449 +0000 UTC m=+21.745274156"
Sep 16 12:48:16.036696 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.036535    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:16.037015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.036535    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:16.037015 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:16.036769    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:16.037015 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:16.036847    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:16.037015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.036535    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:16.037015 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:16.036930    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:16.276956 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.276909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d2szc" event={"ID":"839b45b7-c79f-440d-ac6b-6390e71c58b6","Type":"ContainerStarted","Data":"c70a7f6b0f4cc12ce0378de5ea1b9e5eb6d68e55ccf397030503555119d2c5f9"}
Sep 16 12:48:16.279759 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.279737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"5b2a381191cef632469ad5f61e97501dc285e0daa193b5943c2f8beced9b4235"}
Sep 16 12:48:16.279899 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.279802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"bd78d72f3dc84db12d6338fca8db7913f9c918618ab5b30bfcf9564468ec5096"}
Sep 16 12:48:16.279899 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.279847    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"ebcc9014c941f4891df673e6971d2072ed8783b28f738895239df4a49fbc6c8d"}
Sep 16 12:48:16.279899 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.279884    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"7b20c6209830986b139cd01a3c99dfdc10439a573db7987ec10c6ce31945de9c"}
Sep 16 12:48:16.280026 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.279911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"eeba4c262ad7d72e62234a0c0476b993d967ce27de5c687b721c4739785b41f0"}
Sep 16 12:48:16.281078 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.281056    2558 generic.go:356] "Generic (PLEG): container finished" podID="93857f18-85a0-4668-87ab-8c83030b026c" containerID="d5175a22dd04eef0303c0134b1f83f6f713b48c2a1867da49478248932c25bc0" exitCode=0
Sep 16 12:48:16.281161 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.281125    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerDied","Data":"d5175a22dd04eef0303c0134b1f83f6f713b48c2a1867da49478248932c25bc0"}
Sep 16 12:48:16.282535 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.282515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz" event={"ID":"020b3985-61c1-4945-897f-1e10c5224294","Type":"ContainerStarted","Data":"cd8756164506cdded9b625335a82ed0dcab744efb64c424195c24b49973d89c5"}
Sep 16 12:48:16.283908 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.283877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mqv9c" event={"ID":"e23acb97-072a-4d73-9411-6aee7d0b9259","Type":"ContainerStarted","Data":"f01d785bbb6fd380a814a758d18039c2f76469d18be94fe6f2efb0cbf013dc5e"}
Sep 16 12:48:16.298374 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.298295    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d2szc" podStartSLOduration=4.358351343 podStartE2EDuration="22.298280016s" totalImagesPullingTime="17.939928673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:16.298190903 +0000 UTC m=+22.759819600" watchObservedRunningTime="2026-09-16 12:48:16.298280016 +0000 UTC m=+22.759908712"
Sep 16 12:48:16.298752 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.298730    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nn9ht" podStartSLOduration=4.302543255 podStartE2EDuration="22.298721569s" totalImagesPullingTime="17.996178314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:15.304036682 +0000 UTC m=+21.765665379" watchObservedRunningTime="2026-09-16 12:48:16.298721569 +0000 UTC m=+22.760350264"
Sep 16 12:48:16.370694 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.370450    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mqv9c" podStartSLOduration=4.414959798 podStartE2EDuration="22.370430482s" totalImagesPullingTime="17.955470684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:16.351940265 +0000 UTC m=+22.813568971" watchObservedRunningTime="2026-09-16 12:48:16.370430482 +0000 UTC m=+22.832059182"
Sep 16 12:48:16.654686 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:16.654665    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:48:17.016295 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.016128    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:48:16.654680965Z","UUID":"f83a56ee-1b27-469f-90b9-5050a0a70dd4","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:48:17.017837 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.017794    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:48:17.017971 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.017849    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:48:17.289305 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.289205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz" event={"ID":"020b3985-61c1-4945-897f-1e10c5224294","Type":"ContainerStarted","Data":"53a4780902d3bbf7577cc024807d2016944540eb59d0a604c7f81841a17e0649"}
Sep 16 12:48:17.291503 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.291466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qrml" event={"ID":"be79be6c-655b-4a79-be76-7fb70374c8c6","Type":"ContainerStarted","Data":"b0180525ad2be18f410c8412d2c2e3b584998e94bdbc58da9daf60d8467abf53"}
Sep 16 12:48:17.309725 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.309664    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7qrml" podStartSLOduration=5.3535196769999995 podStartE2EDuration="23.309643929s" totalImagesPullingTime="17.956124252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:17.309096348 +0000 UTC m=+23.770725047" watchObservedRunningTime="2026-09-16 12:48:17.309643929 +0000 UTC m=+23.771272626"
Sep 16 12:48:17.310339 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.310300    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-njp7q" podStartSLOduration=8.855419061 podStartE2EDuration="23.31028568s" totalImagesPullingTime="14.454866619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:16.370769139 +0000 UTC m=+22.832397817" watchObservedRunningTime="2026-09-16 12:48:17.31028568 +0000 UTC m=+23.771914377"
Sep 16 12:48:17.642364 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.642251    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:48:17.642941 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:17.642921    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:48:18.037192 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.036793    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:18.037192 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.036801    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:18.037192 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:18.036943    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:18.037192 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.036987    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:18.037192 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:18.037055    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:18.037192 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:18.037122    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:18.298224 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.298136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"e5460b398201020f1cfa7a9d43fa096e1f2ad10e14b9530ee933b39ccb5a6c11"}
Sep 16 12:48:18.300530 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.300491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz" event={"ID":"020b3985-61c1-4945-897f-1e10c5224294","Type":"ContainerStarted","Data":"42605345fea77d0c5e42e8bf693408814d675a7aa4aaa79649c3c0f339cbc6e7"}
Sep 16 12:48:18.319374 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.319330    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx4bz" podStartSLOduration=3.992794933 podStartE2EDuration="24.319312748s" totalImagesPullingTime="20.326517815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:18.319119676 +0000 UTC m=+24.780748373" watchObservedRunningTime="2026-09-16 12:48:18.319312748 +0000 UTC m=+24.780941443"
Sep 16 12:48:18.926447 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.926411    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:48:18.927018 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:18.927001    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-njp7q"
Sep 16 12:48:20.036443 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:20.036401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:20.036904 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:20.036401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:20.036904 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:20.036517    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:20.036904 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:20.036626    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:20.036904 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:20.036410    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:20.036904 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:20.036718    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:21.310464 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:21.310272    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" event={"ID":"551e18e6-7f02-42aa-8b5f-2132d9df8fcd","Type":"ContainerStarted","Data":"e58b22edcbed3c15c7fba1b0862f764f3f327bcfb2235e442ff91c9137a7bcae"}
Sep 16 12:48:21.311262 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:21.310598    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:48:21.312154 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:21.312131    2558 generic.go:356] "Generic (PLEG): container finished" podID="93857f18-85a0-4668-87ab-8c83030b026c" containerID="032e7c398b83f3cb5b2f8b0dd9a2f762ae4eb8f3e3e38ea24dd49ef471477929" exitCode=0
Sep 16 12:48:21.312261 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:21.312164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerDied","Data":"032e7c398b83f3cb5b2f8b0dd9a2f762ae4eb8f3e3e38ea24dd49ef471477929"}
Sep 16 12:48:21.326245 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:21.326223    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:48:21.341471 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:21.341422    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wb288" podStartSLOduration=9.335863063 podStartE2EDuration="27.341409518s" totalImagesPullingTime="18.005546455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:21.340914622 +0000 UTC m=+27.802543320" watchObservedRunningTime="2026-09-16 12:48:21.341409518 +0000 UTC m=+27.803038244"
Sep 16 12:48:22.037254 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.037027    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:22.037564 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.037030    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:22.037564 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:22.037357    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:22.037564 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.037029    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:22.037564 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:22.037422    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:22.037564 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:22.037541    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:22.315011 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.314982    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:48:22.315396 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.315062    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:48:22.329300 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.329278    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:48:22.451835 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.451792    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2mddz"]
Sep 16 12:48:22.451992 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.451950    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:22.452059 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:22.452039    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:22.455140 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.455115    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs6rn"]
Sep 16 12:48:22.455299 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.455224    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:22.455362 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:22.455320    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:22.473903 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.473879    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cblhw"]
Sep 16 12:48:22.474030 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:22.474006    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:22.474170 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:22.474151    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:23.320767 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:23.320733    2558 generic.go:356] "Generic (PLEG): container finished" podID="93857f18-85a0-4668-87ab-8c83030b026c" containerID="abc3b8b8730d8653db51da58a8c5534aeda934c3e63c1d6f562f9aebd2e1e2fe" exitCode=0
Sep 16 12:48:23.321140 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:23.320797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerDied","Data":"abc3b8b8730d8653db51da58a8c5534aeda934c3e63c1d6f562f9aebd2e1e2fe"}
Sep 16 12:48:24.037519 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:24.037498    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:24.037655 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:24.037590    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:24.037655 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:24.037627    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:24.037758 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:24.037676    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:24.037758 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:24.037687    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:24.037870 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:24.037795    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:24.325957 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:24.325876    2558 generic.go:356] "Generic (PLEG): container finished" podID="93857f18-85a0-4668-87ab-8c83030b026c" containerID="8324cbbfc0f523879aba5a2782846f41e42a6adf14ada7413810d5e97eec1a40" exitCode=0
Sep 16 12:48:24.326291 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:24.325979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerDied","Data":"8324cbbfc0f523879aba5a2782846f41e42a6adf14ada7413810d5e97eec1a40"}
Sep 16 12:48:26.037568 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:26.037533    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:26.038101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:26.037677    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mddz" podUID="8a48d3d0-74d6-4512-8a11-e79ca407a8e4"
Sep 16 12:48:26.038101 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:26.037765    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:26.038101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:26.037906    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cblhw" podUID="9d28d3da-9cae-48ec-ac36-003b8aa41bb6"
Sep 16 12:48:26.038101 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:26.037969    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:26.038101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:26.038048    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rn" podUID="3b346991-b5d0-4414-b808-6380a553c4a5"
Sep 16 12:48:27.804614 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.804580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.804627    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.804687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804761    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804790    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804796    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804803    2558 projected.go:196] Error preparing data for projected volume kube-api-access-6jf8k for pod openshift-network-diagnostics/network-check-target-qs6rn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804796    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804878    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:59.804858625 +0000 UTC m=+66.266487319 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804900    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k podName:3b346991-b5d0-4414-b808-6380a553c4a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:59.804890299 +0000 UTC m=+66.266518973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jf8k" (UniqueName: "kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k") pod "network-check-target-qs6rn" (UID: "3b346991-b5d0-4414-b808-6380a553c4a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:27.805101 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:27.804916    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret podName:8a48d3d0-74d6-4512-8a11-e79ca407a8e4 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:59.804907434 +0000 UTC m=+66.266536109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret") pod "global-pull-secret-syncer-2mddz" (UID: "8a48d3d0-74d6-4512-8a11-e79ca407a8e4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:27.809451 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.809432    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-96.ec2.internal" event="NodeReady"
Sep 16 12:48:27.809594 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.809581    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:48:27.851174 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.851107    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84c8545869-j7bwr"]
Sep 16 12:48:27.919825 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.919775    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gc2qg"]
Sep 16 12:48:27.919995 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.919897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:27.922937 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.922910    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7lm6g\""
Sep 16 12:48:27.923693 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.923674    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:48:27.923807 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.923696    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:48:27.924395 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.924377    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:48:27.930085 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.930063    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:48:27.934690 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.934673    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84c8545869-j7bwr"]
Sep 16 12:48:27.934774 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.934706    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n4xkz"]
Sep 16 12:48:27.934836 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.934781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:27.938801 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.938778    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:48:27.938905 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.938789    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:48:27.938905 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.938898    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sl6fp\""
Sep 16 12:48:27.953577 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.953559    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gc2qg"]
Sep 16 12:48:27.953669 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.953592    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n4xkz"]
Sep 16 12:48:27.953719 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.953672    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:27.957625 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.957608    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-54qld\""
Sep 16 12:48:27.957759 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.957736    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:48:27.957884 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.957843    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:48:27.957931 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:27.957906    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:48:28.007116 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nnvc\" (UniqueName: \"kubernetes.io/projected/96043e4b-3047-471a-909f-4d1e0769cc57-kube-api-access-4nnvc\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.007266 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-certificates\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007266 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007167    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96043e4b-3047-471a-909f-4d1e0769cc57-tmp-dir\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.007266 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007197    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.007266 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-ca-trust-extracted\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007266 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007246    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-installation-pull-secrets\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007486 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-image-registry-private-configuration\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007486 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007303    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007486 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007328    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjq5r\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-kube-api-access-sjq5r\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007486 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007409    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-trusted-ca\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007486 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-bound-sa-token\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.007486 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.007484    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96043e4b-3047-471a-909f-4d1e0769cc57-config-volume\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.036779 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.036747    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:28.036969 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.036791    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:28.036969 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.036753    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:28.039671 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.039650    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:28.039847 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.039799    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:28.040452 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.040436    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djtjx\""
Sep 16 12:48:28.040767 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.040717    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:28.040767 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.040754    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27x5j\""
Sep 16 12:48:28.041065 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.041048    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:28.086653 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.086624    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:28.108733 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.108661    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96043e4b-3047-471a-909f-4d1e0769cc57-config-volume\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.108875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.108839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nnvc\" (UniqueName: \"kubernetes.io/projected/96043e4b-3047-471a-909f-4d1e0769cc57-kube-api-access-4nnvc\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.108875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.108869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-certificates\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.108995 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.108902    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96043e4b-3047-471a-909f-4d1e0769cc57-tmp-dir\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.109049 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnx4m\" (UniqueName: \"kubernetes.io/projected/c0f3be5c-61bb-4f1a-be5f-18484d33e0f4-kube-api-access-hnx4m\") pod \"ingress-canary-n4xkz\" (UID: \"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4\") " pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:28.109101 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.109160 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-ca-trust-extracted\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109160 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-installation-pull-secrets\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-image-registry-private-configuration\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96043e4b-3047-471a-909f-4d1e0769cc57-tmp-dir\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.109280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109227    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109280 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.109238    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:28.109280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjq5r\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-kube-api-access-sjq5r\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109514 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.109294    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls podName:96043e4b-3047-471a-909f-4d1e0769cc57 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:28.609276579 +0000 UTC m=+35.070905257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls") pod "dns-default-gc2qg" (UID: "96043e4b-3047-471a-909f-4d1e0769cc57") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:28.109514 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f3be5c-61bb-4f1a-be5f-18484d33e0f4-cert\") pod \"ingress-canary-n4xkz\" (UID: \"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4\") " pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:28.109514 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-trusted-ca\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109514 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-bound-sa-token\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109514 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109408    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96043e4b-3047-471a-909f-4d1e0769cc57-config-volume\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.109514 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109481    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-certificates\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109806 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.109552    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-ca-trust-extracted\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.109806 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.109611    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:28.109806 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.109624    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c8545869-j7bwr: secret "image-registry-tls" not found
Sep 16 12:48:28.109806 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.109712    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls podName:1dd37e4c-4a3e-46f0-80d6-6c707af9e939 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:28.609694302 +0000 UTC m=+35.071322977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls") pod "image-registry-84c8545869-j7bwr" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939") : secret "image-registry-tls" not found
Sep 16 12:48:28.110505 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.110482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-trusted-ca\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.113365 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.113251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-image-registry-private-configuration\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.113451 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.113295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-installation-pull-secrets\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.122205 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.122181    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nnvc\" (UniqueName: \"kubernetes.io/projected/96043e4b-3047-471a-909f-4d1e0769cc57-kube-api-access-4nnvc\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.122355 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.122336    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjq5r\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-kube-api-access-sjq5r\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.122649 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.122630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-bound-sa-token\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.210095 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.210055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnx4m\" (UniqueName: \"kubernetes.io/projected/c0f3be5c-61bb-4f1a-be5f-18484d33e0f4-kube-api-access-hnx4m\") pod \"ingress-canary-n4xkz\" (UID: \"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4\") " pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:28.210278 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.210140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f3be5c-61bb-4f1a-be5f-18484d33e0f4-cert\") pod \"ingress-canary-n4xkz\" (UID: \"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4\") " pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:28.212792 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.212765    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f3be5c-61bb-4f1a-be5f-18484d33e0f4-cert\") pod \"ingress-canary-n4xkz\" (UID: \"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4\") " pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:28.220075 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.220052    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnx4m\" (UniqueName: \"kubernetes.io/projected/c0f3be5c-61bb-4f1a-be5f-18484d33e0f4-kube-api-access-hnx4m\") pod \"ingress-canary-n4xkz\" (UID: \"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4\") " pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:28.262078 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.262042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n4xkz"
Sep 16 12:48:28.612936 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.612889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:28.612936 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.612935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:28.613146 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.613028    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:28.613146 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.613039    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c8545869-j7bwr: secret "image-registry-tls" not found
Sep 16 12:48:28.613146 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.613091    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls podName:1dd37e4c-4a3e-46f0-80d6-6c707af9e939 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:29.613078314 +0000 UTC m=+36.074706988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls") pod "image-registry-84c8545869-j7bwr" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939") : secret "image-registry-tls" not found
Sep 16 12:48:28.613146 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.613037    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:28.613146 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:28.613135    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls podName:96043e4b-3047-471a-909f-4d1e0769cc57 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:29.613125766 +0000 UTC m=+36.074754454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls") pod "dns-default-gc2qg" (UID: "96043e4b-3047-471a-909f-4d1e0769cc57") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:28.707226 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:28.707159    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n4xkz"]
Sep 16 12:48:28.710431 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:48:28.710405    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f3be5c_61bb_4f1a_be5f_18484d33e0f4.slice/crio-e63bc563075028ba9bc998dc6228cf7477e89795eeac69e681665bac81af4ac0 WatchSource:0}: Error finding container e63bc563075028ba9bc998dc6228cf7477e89795eeac69e681665bac81af4ac0: Status 404 returned error can't find the container with id e63bc563075028ba9bc998dc6228cf7477e89795eeac69e681665bac81af4ac0
Sep 16 12:48:29.344258 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:29.344175    2558 generic.go:356] "Generic (PLEG): container finished" podID="93857f18-85a0-4668-87ab-8c83030b026c" containerID="eb813604f0e3b2966f7e0ee7489aa1a7a5c2c36d6fede40fc46bc020aa279755" exitCode=0
Sep 16 12:48:29.345114 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:29.344278    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerDied","Data":"eb813604f0e3b2966f7e0ee7489aa1a7a5c2c36d6fede40fc46bc020aa279755"}
Sep 16 12:48:29.345471 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:29.345442    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n4xkz" event={"ID":"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4","Type":"ContainerStarted","Data":"e63bc563075028ba9bc998dc6228cf7477e89795eeac69e681665bac81af4ac0"}
Sep 16 12:48:29.621456 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:29.621375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:29.621605 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:29.621498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:29.621605 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:29.621518    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:29.621605 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:29.621544    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c8545869-j7bwr: secret "image-registry-tls" not found
Sep 16 12:48:29.621605 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:29.621601    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:29.621798 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:29.621615    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls podName:1dd37e4c-4a3e-46f0-80d6-6c707af9e939 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:31.621599615 +0000 UTC m=+38.083228289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls") pod "image-registry-84c8545869-j7bwr" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939") : secret "image-registry-tls" not found
Sep 16 12:48:29.621798 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:29.621651    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls podName:96043e4b-3047-471a-909f-4d1e0769cc57 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:31.621635871 +0000 UTC m=+38.083264557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls") pod "dns-default-gc2qg" (UID: "96043e4b-3047-471a-909f-4d1e0769cc57") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:30.351553 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:30.351515    2558 generic.go:356] "Generic (PLEG): container finished" podID="93857f18-85a0-4668-87ab-8c83030b026c" containerID="83e4e0d2bc67b6a918b3a3210eebc13c16b5eabad78c86854bb3ffabd7f49139" exitCode=0
Sep 16 12:48:30.351943 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:30.351583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerDied","Data":"83e4e0d2bc67b6a918b3a3210eebc13c16b5eabad78c86854bb3ffabd7f49139"}
Sep 16 12:48:30.408897 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:30.408870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:31.357593 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.357487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tt472" event={"ID":"93857f18-85a0-4668-87ab-8c83030b026c","Type":"ContainerStarted","Data":"f4c213c9cc4a8313b1d6aa63c8b3e98523594f4c8367a19c72448b0ced8ee200"}
Sep 16 12:48:31.359084 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.359060    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n4xkz" event={"ID":"c0f3be5c-61bb-4f1a-be5f-18484d33e0f4","Type":"ContainerStarted","Data":"cf1a1d1e5f4d3a2862077e5967a8763a549aa1a14853b70bb6bc0ca4452c2bb3"}
Sep 16 12:48:31.382666 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.382617    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tt472" podStartSLOduration=9.302334947 podStartE2EDuration="37.382603973s" totalImagesPullingTime="28.080269026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:31.380835892 +0000 UTC m=+37.842464579" watchObservedRunningTime="2026-09-16 12:48:31.382603973 +0000 UTC m=+37.844232669"
Sep 16 12:48:31.402892 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.402857    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n4xkz" podStartSLOduration=2.057960923 podStartE2EDuration="4.402847784s" totalImagesPullingTime="2.344886861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:31.402027559 +0000 UTC m=+37.863656255" watchObservedRunningTime="2026-09-16 12:48:31.402847784 +0000 UTC m=+37.864476480"
Sep 16 12:48:31.615066 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.614988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:31.638010 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.637987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:31.638097 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.638027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:31.638138 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:31.638127    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:31.638168 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:31.638138    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c8545869-j7bwr: secret "image-registry-tls" not found
Sep 16 12:48:31.638200 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:31.638184    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls podName:1dd37e4c-4a3e-46f0-80d6-6c707af9e939 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:35.638168501 +0000 UTC m=+42.099797174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls") pod "image-registry-84c8545869-j7bwr" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939") : secret "image-registry-tls" not found
Sep 16 12:48:31.638239 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:31.638224    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:31.638280 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:31.638270    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls podName:96043e4b-3047-471a-909f-4d1e0769cc57 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:35.638258028 +0000 UTC m=+42.099886708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls") pod "dns-default-gc2qg" (UID: "96043e4b-3047-471a-909f-4d1e0769cc57") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:31.715449 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.715420    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:31.909189 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:31.909161    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:33.182308 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:33.182276    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:33.280050 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:33.280014    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:34.000831 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:34.000791    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:34.094661 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:34.094625    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:35.668526 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:35.668488    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:35.668958 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:35.668538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:35.668958 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:35.668631    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:35.668958 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:35.668647    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c8545869-j7bwr: secret "image-registry-tls" not found
Sep 16 12:48:35.668958 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:35.668691    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls podName:1dd37e4c-4a3e-46f0-80d6-6c707af9e939 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:43.668678514 +0000 UTC m=+50.130307187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls") pod "image-registry-84c8545869-j7bwr" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939") : secret "image-registry-tls" not found
Sep 16 12:48:35.668958 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:35.668631    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:35.668958 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:35.668766    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls podName:96043e4b-3047-471a-909f-4d1e0769cc57 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:43.668752553 +0000 UTC m=+50.130381227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls") pod "dns-default-gc2qg" (UID: "96043e4b-3047-471a-909f-4d1e0769cc57") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:43.726075 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:43.726029    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:43.726075 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:43.726077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:43.726520 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:43.726176    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:43.726520 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:43.726191    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c8545869-j7bwr: secret "image-registry-tls" not found
Sep 16 12:48:43.726520 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:43.726236    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls podName:1dd37e4c-4a3e-46f0-80d6-6c707af9e939 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:59.726224179 +0000 UTC m=+66.187852852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls") pod "image-registry-84c8545869-j7bwr" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939") : secret "image-registry-tls" not found
Sep 16 12:48:43.726520 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:43.726174    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:43.726520 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:43.726326    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls podName:96043e4b-3047-471a-909f-4d1e0769cc57 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:59.726309017 +0000 UTC m=+66.187937710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls") pod "dns-default-gc2qg" (UID: "96043e4b-3047-471a-909f-4d1e0769cc57") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:54.336548 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:54.336521    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wb288"
Sep 16 12:48:59.743396 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.743356    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:48:59.743766 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.743463    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:48:59.743766 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:59.743519    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:59.743766 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:59.743537    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c8545869-j7bwr: secret "image-registry-tls" not found
Sep 16 12:48:59.743766 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:59.743551    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:59.743766 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:59.743603    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls podName:96043e4b-3047-471a-909f-4d1e0769cc57 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:31.743587006 +0000 UTC m=+98.205215680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls") pod "dns-default-gc2qg" (UID: "96043e4b-3047-471a-909f-4d1e0769cc57") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:59.743766 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:59.743619    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls podName:1dd37e4c-4a3e-46f0-80d6-6c707af9e939 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:31.743611376 +0000 UTC m=+98.205240050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls") pod "image-registry-84c8545869-j7bwr" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939") : secret "image-registry-tls" not found
Sep 16 12:48:59.844773 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.844732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:48:59.844999 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.844781    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:48:59.844999 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.844927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:59.847783 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.847763    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:59.847898 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.847796    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:59.847898 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.847880    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:59.855562 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:59.855544    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:48:59.855636 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:48:59.855609    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs podName:9d28d3da-9cae-48ec-ac36-003b8aa41bb6 nodeName:}" failed. No retries permitted until 2026-09-16 12:50:03.85558848 +0000 UTC m=+130.317217155 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs") pod "network-metrics-daemon-cblhw" (UID: "9d28d3da-9cae-48ec-ac36-003b8aa41bb6") : secret "metrics-daemon-secret" not found
Sep 16 12:48:59.858273 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.858259    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:59.858550 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.858530    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a48d3d0-74d6-4512-8a11-e79ca407a8e4-original-pull-secret\") pod \"global-pull-secret-syncer-2mddz\" (UID: \"8a48d3d0-74d6-4512-8a11-e79ca407a8e4\") " pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:59.862691 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.862674    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mddz"
Sep 16 12:48:59.867882 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:48:59.867859    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf8k\" (UniqueName: \"kubernetes.io/projected/3b346991-b5d0-4414-b808-6380a553c4a5-kube-api-access-6jf8k\") pod \"network-check-target-qs6rn\" (UID: \"3b346991-b5d0-4414-b808-6380a553c4a5\") " pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:49:00.132564 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:00.132496    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2mddz"]
Sep 16 12:49:00.135874 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:00.135847    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a48d3d0_74d6_4512_8a11_e79ca407a8e4.slice/crio-b782fd29a914d879a5f7474834663a6bfcbffc7909492b064040243296e499d7 WatchSource:0}: Error finding container b782fd29a914d879a5f7474834663a6bfcbffc7909492b064040243296e499d7: Status 404 returned error can't find the container with id b782fd29a914d879a5f7474834663a6bfcbffc7909492b064040243296e499d7
Sep 16 12:49:00.158252 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:00.158229    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djtjx\""
Sep 16 12:49:00.165786 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:00.165761    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:49:00.431570 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:00.431539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2mddz" event={"ID":"8a48d3d0-74d6-4512-8a11-e79ca407a8e4","Type":"ContainerStarted","Data":"b782fd29a914d879a5f7474834663a6bfcbffc7909492b064040243296e499d7"}
Sep 16 12:49:00.437963 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:00.437936    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs6rn"]
Sep 16 12:49:00.441165 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:00.441144    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b346991_b5d0_4414_b808_6380a553c4a5.slice/crio-6fb0df79235b223efba1d2ea5ddc0ae23f54591353436a49c60ac815248139f7 WatchSource:0}: Error finding container 6fb0df79235b223efba1d2ea5ddc0ae23f54591353436a49c60ac815248139f7: Status 404 returned error can't find the container with id 6fb0df79235b223efba1d2ea5ddc0ae23f54591353436a49c60ac815248139f7
Sep 16 12:49:01.435588 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:01.435555    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs6rn" event={"ID":"3b346991-b5d0-4414-b808-6380a553c4a5","Type":"ContainerStarted","Data":"6fb0df79235b223efba1d2ea5ddc0ae23f54591353436a49c60ac815248139f7"}
Sep 16 12:49:05.446519 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.446482    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2mddz" event={"ID":"8a48d3d0-74d6-4512-8a11-e79ca407a8e4","Type":"ContainerStarted","Data":"abcbbcc399888f24b70b3164643c8b493595ef7dd2bf7c58b9a7dab29eed3056"}
Sep 16 12:49:05.447900 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.447875    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs6rn" event={"ID":"3b346991-b5d0-4414-b808-6380a553c4a5","Type":"ContainerStarted","Data":"39b573b3791a1c6020a8002e204a6d5519f482df7d566b8c176fb1c610df697f"}
Sep 16 12:49:05.448014 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.447994    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:49:05.473445 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.473390    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2mddz" podStartSLOduration=66.858822811 podStartE2EDuration="1m11.473375983s" totalImagesPullingTime="4.614553172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:05.47215351 +0000 UTC m=+71.933782203" watchObservedRunningTime="2026-09-16 12:49:05.473375983 +0000 UTC m=+71.935004669"
Sep 16 12:49:05.490059 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.490020    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qs6rn" podStartSLOduration=67.184224866 podStartE2EDuration="1m11.490007535s" totalImagesPullingTime="4.305782669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:05.489270575 +0000 UTC m=+71.950899272" watchObservedRunningTime="2026-09-16 12:49:05.490007535 +0000 UTC m=+71.951636253"
Sep 16 12:49:05.741621 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.741546    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ddbbdd75d-mdklm"]
Sep 16 12:49:05.745188 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.745172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.752300 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.752268    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:49:05.752444 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.752355    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-95x46\""
Sep 16 12:49:05.752444 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.752368    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 12:49:05.752552 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.752504    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:49:05.752605 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.752591    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 12:49:05.753553 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.753533    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 12:49:05.753674 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.753591    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 12:49:05.763075 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.763055    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddbbdd75d-mdklm"]
Sep 16 12:49:05.782463 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.782437    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-default-certificate\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.782536 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.782468    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-stats-auth\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.782536 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.782507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2j5\" (UniqueName: \"kubernetes.io/projected/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-kube-api-access-qw2j5\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.782536 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.782532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.782641 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.782551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.882842 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.882798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-stats-auth\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.882988 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.882878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2j5\" (UniqueName: \"kubernetes.io/projected/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-kube-api-access-qw2j5\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.882988 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.882908    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.882988 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.882934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.883105 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.882997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-default-certificate\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.883105 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:05.883060    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:06.383042988 +0000 UTC m=+72.844671667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:05.883105 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:05.883087    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:05.883224 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:05.883139    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:06.383122555 +0000 UTC m=+72.844751244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : secret "router-metrics-certs-default" not found
Sep 16 12:49:05.885282 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.885261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-default-certificate\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.885383 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.885300    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-stats-auth\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:05.892679 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:05.892657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2j5\" (UniqueName: \"kubernetes.io/projected/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-kube-api-access-qw2j5\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:06.386942 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:06.386889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:06.386942 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:06.386943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:06.387166 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:06.387072    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:07.387052671 +0000 UTC m=+73.848681344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:06.387166 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:06.387096    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:06.387166 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:06.387154    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:07.387142955 +0000 UTC m=+73.848771645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : secret "router-metrics-certs-default" not found
Sep 16 12:49:07.394204 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:07.394169    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:07.394204 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:07.394204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:07.394622 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:07.394292    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:07.394622 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:07.394328    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:09.394311726 +0000 UTC m=+75.855940419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:07.394622 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:07.394349    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:09.394343074 +0000 UTC m=+75.855971747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : secret "router-metrics-certs-default" not found
Sep 16 12:49:09.409040 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:09.409005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:09.409040 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:09.409038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:09.409518 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:09.409214    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.409188542 +0000 UTC m=+79.870817231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:09.409518 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:09.409224    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:09.409518 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:09.409268    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.409258043 +0000 UTC m=+79.870886737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : secret "router-metrics-certs-default" not found
Sep 16 12:49:12.635344 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.635305    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-np6cj"]
Sep 16 12:49:12.638916 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.638892    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.641855 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.641827    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:49:12.641973 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.641825    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:49:12.641973 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.641891    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x6fmq\""
Sep 16 12:49:12.643167 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.643147    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:49:12.643269 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.643198    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:49:12.652172 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.652149    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-np6cj"]
Sep 16 12:49:12.733659 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.733625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d8b0d29-3267-449c-a4f7-784bfe0113fb-data-volume\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.733844 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.733686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnrn\" (UniqueName: \"kubernetes.io/projected/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-api-access-6xnrn\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.733844 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.733713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.733844 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.733783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d8b0d29-3267-449c-a4f7-784bfe0113fb-crio-socket\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.733844 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.733829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.835178 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.835137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnrn\" (UniqueName: \"kubernetes.io/projected/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-api-access-6xnrn\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.835356 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.835193    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.835356 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.835242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d8b0d29-3267-449c-a4f7-784bfe0113fb-crio-socket\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.835356 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.835269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.835356 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:12.835327    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:12.835553 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.835364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d8b0d29-3267-449c-a4f7-784bfe0113fb-crio-socket\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.835553 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.835334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d8b0d29-3267-449c-a4f7-784bfe0113fb-data-volume\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.835553 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:12.835405    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.335383574 +0000 UTC m=+79.797012251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:12.835553 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:12.835442    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:12.835553 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:12.835503    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.335490116 +0000 UTC m=+79.797118795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:12.835770 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.835667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d8b0d29-3267-449c-a4f7-784bfe0113fb-data-volume\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:12.845545 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:12.845517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnrn\" (UniqueName: \"kubernetes.io/projected/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-api-access-6xnrn\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:13.339136 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:13.339097    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:13.339315 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:13.339151    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:13.339315 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:13.339220    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:13.339315 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:13.339235    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:13.339315 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:13.339293    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.339277729 +0000 UTC m=+80.800906407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:13.339315 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:13.339313    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.339306216 +0000 UTC m=+80.800934889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:13.440267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:13.440230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:13.440267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:13.440267    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:13.440472 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:13.440392    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:13.440472 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:13.440420    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:21.440400802 +0000 UTC m=+87.902029496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:13.440472 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:13.440444    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:21.440436375 +0000 UTC m=+87.902065049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : secret "router-metrics-certs-default" not found
Sep 16 12:49:14.347374 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:14.347334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:14.347760 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:14.347387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:14.347760 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:14.347454    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:14.347760 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:14.347475    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:14.347760 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:14.347517    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:16.347502291 +0000 UTC m=+82.809130966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:14.347760 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:14.347532    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:16.347525922 +0000 UTC m=+82.809154595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:15.391118 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:15.391090    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d2szc_839b45b7-c79f-440d-ac6b-6390e71c58b6/dns-node-resolver/0.log"
Sep 16 12:49:16.364352 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:16.364308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:16.364522 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:16.364458    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:16.364522 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:16.364493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:16.364522 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:16.364517    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.36450102 +0000 UTC m=+86.826129694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:16.364652 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:16.364546    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:16.364652 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:16.364599    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.364584642 +0000 UTC m=+86.826213332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:16.603360 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:16.603265    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mqv9c_e23acb97-072a-4d73-9411-6aee7d0b9259/node-ca/0.log"
Sep 16 12:49:17.231713 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:17.231659    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n4xkz_c0f3be5c-61bb-4f1a-be5f-18484d33e0f4/serve-healthcheck-canary/0.log"
Sep 16 12:49:20.393343 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:20.393312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:20.393702 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:20.393367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:20.393702 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:20.393444    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:20.393702 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:20.393506    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:28.393489429 +0000 UTC m=+94.855118117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:20.393702 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:20.393512    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:20.393702 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:20.393545    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls podName:3d8b0d29-3267-449c-a4f7-784bfe0113fb nodeName:}" failed. No retries permitted until 2026-09-16 12:49:28.393535292 +0000 UTC m=+94.855163979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-np6cj" (UID: "3d8b0d29-3267-449c-a4f7-784bfe0113fb") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:21.502038 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:21.502004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:21.502423 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:21.502130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:21.502423 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:21.502236    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle podName:f9e9ee04-dd2e-492e-9a74-1ef0d951cbff nodeName:}" failed. No retries permitted until 2026-09-16 12:49:37.50222296 +0000 UTC m=+103.963851633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle") pod "router-default-ddbbdd75d-mdklm" (UID: "f9e9ee04-dd2e-492e-9a74-1ef0d951cbff") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:21.504328 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:21.504308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-metrics-certs\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:28.453069 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:28.453026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:28.453473 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:28.453092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:28.453555 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:28.453534    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d8b0d29-3267-449c-a4f7-784bfe0113fb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:28.455355 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:28.455335    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d8b0d29-3267-449c-a4f7-784bfe0113fb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-np6cj\" (UID: \"3d8b0d29-3267-449c-a4f7-784bfe0113fb\") " pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:28.548438 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:28.548403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-np6cj"
Sep 16 12:49:28.831370 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:28.831297    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-np6cj"]
Sep 16 12:49:28.834796 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:28.834765    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8b0d29_3267_449c_a4f7_784bfe0113fb.slice/crio-8e908b77c90689711cdc4e7e91abe092082fb88e041d790bc880c05d26bd3827 WatchSource:0}: Error finding container 8e908b77c90689711cdc4e7e91abe092082fb88e041d790bc880c05d26bd3827: Status 404 returned error can't find the container with id 8e908b77c90689711cdc4e7e91abe092082fb88e041d790bc880c05d26bd3827
Sep 16 12:49:29.508931 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:29.508901    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np6cj" event={"ID":"3d8b0d29-3267-449c-a4f7-784bfe0113fb","Type":"ContainerStarted","Data":"8a1642712fe02636d4a53fd5989435d63de3081ddd7a1fd23ddd25c9e90887d8"}
Sep 16 12:49:29.509191 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:29.508982    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np6cj" event={"ID":"3d8b0d29-3267-449c-a4f7-784bfe0113fb","Type":"ContainerStarted","Data":"15153087c7208fe1393983459c8de3d94a8723f4a24bb3f289d851896e326615"}
Sep 16 12:49:29.509191 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:29.509001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np6cj" event={"ID":"3d8b0d29-3267-449c-a4f7-784bfe0113fb","Type":"ContainerStarted","Data":"8e908b77c90689711cdc4e7e91abe092082fb88e041d790bc880c05d26bd3827"}
Sep 16 12:49:31.517376 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.517336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-np6cj" event={"ID":"3d8b0d29-3267-449c-a4f7-784bfe0113fb","Type":"ContainerStarted","Data":"a97a46512d9edfdb25921dc912631f99fa6892ff5552c6fe56f86675b215d0d7"}
Sep 16 12:49:31.543322 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.543273    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-np6cj" podStartSLOduration=17.714771888 podStartE2EDuration="19.54325958s" totalImagesPullingTime="1.828487692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:31.541579963 +0000 UTC m=+98.003208659" watchObservedRunningTime="2026-09-16 12:49:31.54325958 +0000 UTC m=+98.004888275"
Sep 16 12:49:31.779123 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.779030    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:49:31.779123 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.779074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:49:31.781464 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.781431    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96043e4b-3047-471a-909f-4d1e0769cc57-metrics-tls\") pod \"dns-default-gc2qg\" (UID: \"96043e4b-3047-471a-909f-4d1e0769cc57\") " pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:49:31.781575 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.781485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"image-registry-84c8545869-j7bwr\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") " pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:49:31.835779 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.835752    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7lm6g\""
Sep 16 12:49:31.843229 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.843210    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:49:31.847624 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.847606    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sl6fp\""
Sep 16 12:49:31.855349 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:31.855329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:49:32.129293 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:32.129218    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84c8545869-j7bwr"]
Sep 16 12:49:32.133203 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:32.133177    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd37e4c_4a3e_46f0_80d6_6c707af9e939.slice/crio-4ca9bf6d20beee8ef8429b499f79ac66d82eff90119548be73fa5f65902b2918 WatchSource:0}: Error finding container 4ca9bf6d20beee8ef8429b499f79ac66d82eff90119548be73fa5f65902b2918: Status 404 returned error can't find the container with id 4ca9bf6d20beee8ef8429b499f79ac66d82eff90119548be73fa5f65902b2918
Sep 16 12:49:32.147676 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:32.147642    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gc2qg"]
Sep 16 12:49:32.150729 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:32.150707    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96043e4b_3047_471a_909f_4d1e0769cc57.slice/crio-0b16395952d70dafc366dc11deb6079370231bf20de06b54d45fab77dfce3462 WatchSource:0}: Error finding container 0b16395952d70dafc366dc11deb6079370231bf20de06b54d45fab77dfce3462: Status 404 returned error can't find the container with id 0b16395952d70dafc366dc11deb6079370231bf20de06b54d45fab77dfce3462
Sep 16 12:49:32.521319 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:32.521282    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gc2qg" event={"ID":"96043e4b-3047-471a-909f-4d1e0769cc57","Type":"ContainerStarted","Data":"0b16395952d70dafc366dc11deb6079370231bf20de06b54d45fab77dfce3462"}
Sep 16 12:49:32.523118 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:32.523086    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c8545869-j7bwr" event={"ID":"1dd37e4c-4a3e-46f0-80d6-6c707af9e939","Type":"ContainerStarted","Data":"bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841"}
Sep 16 12:49:32.523242 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:32.523134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c8545869-j7bwr" event={"ID":"1dd37e4c-4a3e-46f0-80d6-6c707af9e939","Type":"ContainerStarted","Data":"4ca9bf6d20beee8ef8429b499f79ac66d82eff90119548be73fa5f65902b2918"}
Sep 16 12:49:32.523715 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:32.523696    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:49:32.555298 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:32.555255    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84c8545869-j7bwr" podStartSLOduration=98.555240212 podStartE2EDuration="1m38.555240212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:32.553319979 +0000 UTC m=+99.014948675" watchObservedRunningTime="2026-09-16 12:49:32.555240212 +0000 UTC m=+99.016868909"
Sep 16 12:49:34.531490 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:34.531455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gc2qg" event={"ID":"96043e4b-3047-471a-909f-4d1e0769cc57","Type":"ContainerStarted","Data":"ffaf12e24672f07f819a16ab4bef4e834fb07350741475e9ec4b5ae1da50d1a7"}
Sep 16 12:49:34.531901 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:34.531508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gc2qg" event={"ID":"96043e4b-3047-471a-909f-4d1e0769cc57","Type":"ContainerStarted","Data":"16287f2e2ba4a39ce76d7769a623ee03b47e55fcddc8ee564b69b077d13ed07f"}
Sep 16 12:49:34.531901 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:34.531596    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:49:34.554522 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:34.554466    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gc2qg" podStartSLOduration=66.020864239 podStartE2EDuration="1m7.554451102s" totalImagesPullingTime="1.533586863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:34.552509446 +0000 UTC m=+101.014138147" watchObservedRunningTime="2026-09-16 12:49:34.554451102 +0000 UTC m=+101.016079797"
Sep 16 12:49:36.452869 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:36.452836    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qs6rn"
Sep 16 12:49:37.522312 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:37.522276    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:37.522894 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:37.522876    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e9ee04-dd2e-492e-9a74-1ef0d951cbff-service-ca-bundle\") pod \"router-default-ddbbdd75d-mdklm\" (UID: \"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff\") " pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:37.557357 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:37.557328    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:37.863118 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:37.862952    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddbbdd75d-mdklm"]
Sep 16 12:49:37.865493 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:37.865463    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e9ee04_dd2e_492e_9a74_1ef0d951cbff.slice/crio-674dd1f0aee71c97e8089cf72858f9266a755c2d67631e8b7caa5eaff8da5d87 WatchSource:0}: Error finding container 674dd1f0aee71c97e8089cf72858f9266a755c2d67631e8b7caa5eaff8da5d87: Status 404 returned error can't find the container with id 674dd1f0aee71c97e8089cf72858f9266a755c2d67631e8b7caa5eaff8da5d87
Sep 16 12:49:38.545433 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:38.545399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddbbdd75d-mdklm" event={"ID":"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff","Type":"ContainerStarted","Data":"d2828c46290cf897bc482cbb099178011c75d524d9f87d95ee5b7869aa865bea"}
Sep 16 12:49:38.545889 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:38.545445    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddbbdd75d-mdklm" event={"ID":"f9e9ee04-dd2e-492e-9a74-1ef0d951cbff","Type":"ContainerStarted","Data":"674dd1f0aee71c97e8089cf72858f9266a755c2d67631e8b7caa5eaff8da5d87"}
Sep 16 12:49:38.558390 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:38.558361    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:38.560905 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:38.560885    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:38.592857 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:38.592797    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-ddbbdd75d-mdklm" podStartSLOduration=33.592781492 podStartE2EDuration="33.592781492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:38.570033763 +0000 UTC m=+105.031662462" watchObservedRunningTime="2026-09-16 12:49:38.592781492 +0000 UTC m=+105.054410187"
Sep 16 12:49:39.021792 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.021759    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-c9hsx"]
Sep 16 12:49:39.046211 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.046184    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-c9hsx"]
Sep 16 12:49:39.046350 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.046280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-c9hsx"
Sep 16 12:49:39.051700 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.051676    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:49:39.052042 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.052026    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:49:39.063439 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.063421    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-n8gwp\""
Sep 16 12:49:39.133934 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.133898    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnl77\" (UniqueName: \"kubernetes.io/projected/19f33b29-948e-4d9c-bb45-1e036c46358f-kube-api-access-tnl77\") pod \"downloads-76f99584f4-c9hsx\" (UID: \"19f33b29-948e-4d9c-bb45-1e036c46358f\") " pod="openshift-console/downloads-76f99584f4-c9hsx"
Sep 16 12:49:39.234690 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.234653    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnl77\" (UniqueName: \"kubernetes.io/projected/19f33b29-948e-4d9c-bb45-1e036c46358f-kube-api-access-tnl77\") pod \"downloads-76f99584f4-c9hsx\" (UID: \"19f33b29-948e-4d9c-bb45-1e036c46358f\") " pod="openshift-console/downloads-76f99584f4-c9hsx"
Sep 16 12:49:39.252735 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.252710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnl77\" (UniqueName: \"kubernetes.io/projected/19f33b29-948e-4d9c-bb45-1e036c46358f-kube-api-access-tnl77\") pod \"downloads-76f99584f4-c9hsx\" (UID: \"19f33b29-948e-4d9c-bb45-1e036c46358f\") " pod="openshift-console/downloads-76f99584f4-c9hsx"
Sep 16 12:49:39.354920 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.354844    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-c9hsx"
Sep 16 12:49:39.548823 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.548793    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:39.550129 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.550113    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ddbbdd75d-mdklm"
Sep 16 12:49:39.644108 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.644081    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-c9hsx"]
Sep 16 12:49:39.647430 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:39.647404    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f33b29_948e_4d9c_bb45_1e036c46358f.slice/crio-c3d69d29b99b328102e73f672926f6bae727b42d77a908209d71899e62d2dc83 WatchSource:0}: Error finding container c3d69d29b99b328102e73f672926f6bae727b42d77a908209d71899e62d2dc83: Status 404 returned error can't find the container with id c3d69d29b99b328102e73f672926f6bae727b42d77a908209d71899e62d2dc83
Sep 16 12:49:39.738341 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.738308    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b6676594c-rvb4z"]
Sep 16 12:49:39.768424 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.768394    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6676594c-rvb4z"]
Sep 16 12:49:39.768561 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.768491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.771512 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.771485    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:49:39.771512 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.771503    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:49:39.771689 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.771556    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:49:39.771689 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.771505    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:49:39.771860 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.771850    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z9clh\""
Sep 16 12:49:39.771941 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.771855    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:49:39.839097 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.839069    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-console-config\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.839218 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.839144    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-oauth-serving-cert\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.839218 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.839180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-oauth-config\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.839218 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.839210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-serving-cert\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.839320 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.839271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvdh7\" (UniqueName: \"kubernetes.io/projected/51a4d013-d7de-4582-aeca-74b2accfe812-kube-api-access-mvdh7\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.839320 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.839290    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-service-ca\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.939678 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.939596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-console-config\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.939678 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.939647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-oauth-serving-cert\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.939912 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.939683    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-oauth-config\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.939912 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.939707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-serving-cert\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.939912 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.939740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvdh7\" (UniqueName: \"kubernetes.io/projected/51a4d013-d7de-4582-aeca-74b2accfe812-kube-api-access-mvdh7\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.939912 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.939765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-service-ca\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.940499 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.940473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-console-config\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.940605 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.940522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-service-ca\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.940605 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.940522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-oauth-serving-cert\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.942178 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.942157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-serving-cert\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.942267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.942177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-oauth-config\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:39.949566 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:39.949543    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvdh7\" (UniqueName: \"kubernetes.io/projected/51a4d013-d7de-4582-aeca-74b2accfe812-kube-api-access-mvdh7\") pod \"console-7b6676594c-rvb4z\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") " pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:40.077987 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:40.077952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:40.355316 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:40.355234    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6676594c-rvb4z"]
Sep 16 12:49:40.358413 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:40.358386    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a4d013_d7de_4582_aeca_74b2accfe812.slice/crio-1ab90abd8742a98f81e45519add5800c63a4fa88208bd5aa49b1a737505033a3 WatchSource:0}: Error finding container 1ab90abd8742a98f81e45519add5800c63a4fa88208bd5aa49b1a737505033a3: Status 404 returned error can't find the container with id 1ab90abd8742a98f81e45519add5800c63a4fa88208bd5aa49b1a737505033a3
Sep 16 12:49:40.554383 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:40.554345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6676594c-rvb4z" event={"ID":"51a4d013-d7de-4582-aeca-74b2accfe812","Type":"ContainerStarted","Data":"1ab90abd8742a98f81e45519add5800c63a4fa88208bd5aa49b1a737505033a3"}
Sep 16 12:49:40.555771 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:40.555741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-c9hsx" event={"ID":"19f33b29-948e-4d9c-bb45-1e036c46358f","Type":"ContainerStarted","Data":"c3d69d29b99b328102e73f672926f6bae727b42d77a908209d71899e62d2dc83"}
Sep 16 12:49:44.161356 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.161316    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"]
Sep 16 12:49:44.164630 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.164602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.167457 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.167427    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 12:49:44.167604 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.167575    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 12:49:44.168599 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.168581    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:49:44.168784 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.168762    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:49:44.169009 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.168794    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:49:44.169009 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.168926    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zggtq\""
Sep 16 12:49:44.177960 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.177932    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"]
Sep 16 12:49:44.277588 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.277554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.277588 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.277592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.277805 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.277642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18132788-212f-4427-a048-cce30d147a6d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.277805 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.277780    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbkbv\" (UniqueName: \"kubernetes.io/projected/18132788-212f-4427-a048-cce30d147a6d-kube-api-access-rbkbv\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.379137 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.379099    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.379137 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.379141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.379376 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.379167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18132788-212f-4427-a048-cce30d147a6d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.379376 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.379272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbkbv\" (UniqueName: \"kubernetes.io/projected/18132788-212f-4427-a048-cce30d147a6d-kube-api-access-rbkbv\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.379376 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:44.379317    2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 12:49:44.379503 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:44.379397    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls podName:18132788-212f-4427-a048-cce30d147a6d nodeName:}" failed. No retries permitted until 2026-09-16 12:49:44.879376397 +0000 UTC m=+111.341005073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-zkpf8" (UID: "18132788-212f-4427-a048-cce30d147a6d") : secret "prometheus-operator-tls" not found
Sep 16 12:49:44.381076 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.380434    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18132788-212f-4427-a048-cce30d147a6d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.383767 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.382396    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.391281 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.391260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbkbv\" (UniqueName: \"kubernetes.io/projected/18132788-212f-4427-a048-cce30d147a6d-kube-api-access-rbkbv\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.537377 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.537292    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gc2qg"
Sep 16 12:49:44.573617 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.573577    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6676594c-rvb4z" event={"ID":"51a4d013-d7de-4582-aeca-74b2accfe812","Type":"ContainerStarted","Data":"cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22"}
Sep 16 12:49:44.595339 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.595280    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b6676594c-rvb4z" podStartSLOduration=2.297874059 podStartE2EDuration="5.59526345s" totalImagesPullingTime="3.297389391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:44.594153631 +0000 UTC m=+111.055782328" watchObservedRunningTime="2026-09-16 12:49:44.59526345 +0000 UTC m=+111.056892140"
Sep 16 12:49:44.883392 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:44.883307    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:44.883546 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:44.883447    2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 12:49:44.883546 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:44.883510    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls podName:18132788-212f-4427-a048-cce30d147a6d nodeName:}" failed. No retries permitted until 2026-09-16 12:49:45.88349582 +0000 UTC m=+112.345124494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-zkpf8" (UID: "18132788-212f-4427-a048-cce30d147a6d") : secret "prometheus-operator-tls" not found
Sep 16 12:49:45.892654 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:45.892612    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:45.895222 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:45.895196    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18132788-212f-4427-a048-cce30d147a6d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zkpf8\" (UID: \"18132788-212f-4427-a048-cce30d147a6d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:45.975306 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:45.975273    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"
Sep 16 12:49:46.254772 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:46.254694    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8"]
Sep 16 12:49:46.257952 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:46.257927    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18132788_212f_4427_a048_cce30d147a6d.slice/crio-458974b42958384dea41a192d76e6037f26faa424b51139e91f8aaeaf7773c74 WatchSource:0}: Error finding container 458974b42958384dea41a192d76e6037f26faa424b51139e91f8aaeaf7773c74: Status 404 returned error can't find the container with id 458974b42958384dea41a192d76e6037f26faa424b51139e91f8aaeaf7773c74
Sep 16 12:49:46.581977 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:46.581899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8" event={"ID":"18132788-212f-4427-a048-cce30d147a6d","Type":"ContainerStarted","Data":"458974b42958384dea41a192d76e6037f26faa424b51139e91f8aaeaf7773c74"}
Sep 16 12:49:47.589197 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:47.589166    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8" event={"ID":"18132788-212f-4427-a048-cce30d147a6d","Type":"ContainerStarted","Data":"af0da51ff50c0ffcb7598cc020a7c1163b1aaf8311c4619cf683c15b01620f72"}
Sep 16 12:49:48.116994 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.116959    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65fc54b68f-4z55w"]
Sep 16 12:49:48.120116 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.120095    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.129529 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.129102    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:49:48.133084 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.133051    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fc54b68f-4z55w"]
Sep 16 12:49:48.212884 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.212833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-oauth-config\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.213062 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.212901    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-config\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.213062 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.212919    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-trusted-ca-bundle\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.213062 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.212940    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-oauth-serving-cert\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.213062 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.212968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-serving-cert\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.213233 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.213085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7ph\" (UniqueName: \"kubernetes.io/projected/7581b5ab-919c-4ad2-9141-9f76f2da17f7-kube-api-access-pm7ph\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.213233 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.213130    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-service-ca\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.314147 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.314112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-serving-cert\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.314318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.314177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7ph\" (UniqueName: \"kubernetes.io/projected/7581b5ab-919c-4ad2-9141-9f76f2da17f7-kube-api-access-pm7ph\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.314318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.314196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-service-ca\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.314318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.314229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-oauth-config\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.314318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.314252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-config\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.314318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.314274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-trusted-ca-bundle\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.314842 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.314696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-oauth-serving-cert\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.315156 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.315090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-service-ca\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.315156 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.315144    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-config\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.315297 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.315270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-trusted-ca-bundle\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.315475 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.315454    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-oauth-serving-cert\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.317121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.317087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-serving-cert\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.317238 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.317093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-oauth-config\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.324874 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.324854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7ph\" (UniqueName: \"kubernetes.io/projected/7581b5ab-919c-4ad2-9141-9f76f2da17f7-kube-api-access-pm7ph\") pod \"console-65fc54b68f-4z55w\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") " pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.430227 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.430180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:48.594986 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.594942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8" event={"ID":"18132788-212f-4427-a048-cce30d147a6d","Type":"ContainerStarted","Data":"3b66b45a06bfa7043ad23701d52bbe4a5494f9812a580ee1dfb9de76706e3b91"}
Sep 16 12:49:48.614671 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.614609    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zkpf8" podStartSLOduration=3.367688379 podStartE2EDuration="4.614587753s" totalImagesPullingTime="1.246899374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:48.613541109 +0000 UTC m=+115.075169805" watchObservedRunningTime="2026-09-16 12:49:48.614587753 +0000 UTC m=+115.076216450"
Sep 16 12:49:48.699742 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:48.699669    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fc54b68f-4z55w"]
Sep 16 12:49:48.703238 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:48.703208    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7581b5ab_919c_4ad2_9141_9f76f2da17f7.slice/crio-caa70fbec3fd4dbb664e8ebed01051422e85c3a6da863b61cd5dd6a37c10d71a WatchSource:0}: Error finding container caa70fbec3fd4dbb664e8ebed01051422e85c3a6da863b61cd5dd6a37c10d71a: Status 404 returned error can't find the container with id caa70fbec3fd4dbb664e8ebed01051422e85c3a6da863b61cd5dd6a37c10d71a
Sep 16 12:49:49.599561 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:49.599526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fc54b68f-4z55w" event={"ID":"7581b5ab-919c-4ad2-9141-9f76f2da17f7","Type":"ContainerStarted","Data":"df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da"}
Sep 16 12:49:49.599959 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:49.599581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fc54b68f-4z55w" event={"ID":"7581b5ab-919c-4ad2-9141-9f76f2da17f7","Type":"ContainerStarted","Data":"caa70fbec3fd4dbb664e8ebed01051422e85c3a6da863b61cd5dd6a37c10d71a"}
Sep 16 12:49:49.624417 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:49.624377    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65fc54b68f-4z55w" podStartSLOduration=1.624362657 podStartE2EDuration="1.624362657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:49.622972974 +0000 UTC m=+116.084601672" watchObservedRunningTime="2026-09-16 12:49:49.624362657 +0000 UTC m=+116.085991352"
Sep 16 12:49:50.078481 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.078443    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:50.078649 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.078507    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:50.083960 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.083936    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:50.555212 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.555183    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bgjwx"]
Sep 16 12:49:50.559578 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.559557    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.562318 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.562208    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:49:50.562502 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.562481    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dd2m8\""
Sep 16 12:49:50.562588 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.562511    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:49:50.562637 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.562518    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:49:50.609366 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.609340    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:49:50.633951 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.633916    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffk9\" (UniqueName: \"kubernetes.io/projected/f8cac50e-c94f-4460-902c-5c2e7e3c075d-kube-api-access-nffk9\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634120 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.633965    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-textfile\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634120 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.634004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-wtmp\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634120 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.634028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-tls\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634120 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.634100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-sys\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634346 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.634123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634346 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.634152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-root\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634346 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.634210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cac50e-c94f-4460-902c-5c2e7e3c075d-metrics-client-ca\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.634346 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.634309    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.734766 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.734724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.734965 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.734914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffk9\" (UniqueName: \"kubernetes.io/projected/f8cac50e-c94f-4460-902c-5c2e7e3c075d-kube-api-access-nffk9\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735024 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.734974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-textfile\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735024 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-wtmp\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735130 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-tls\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-sys\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735375 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-root\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735375 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cac50e-c94f-4460-902c-5c2e7e3c075d-metrics-client-ca\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.735480 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735456    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.736418 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.735854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cac50e-c94f-4460-902c-5c2e7e3c075d-metrics-client-ca\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.736546 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.736398    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-sys\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.736546 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.736479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-textfile\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.736546 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.736520    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-root\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.736710 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.736571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-wtmp\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.736710 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:50.736608    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 12:49:50.736710 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:50.736662    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-tls podName:f8cac50e-c94f-4460-902c-5c2e7e3c075d nodeName:}" failed. No retries permitted until 2026-09-16 12:49:51.236643203 +0000 UTC m=+117.698271883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-tls") pod "node-exporter-bgjwx" (UID: "f8cac50e-c94f-4460-902c-5c2e7e3c075d") : secret "node-exporter-tls" not found
Sep 16 12:49:50.738686 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.738662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:50.745634 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:50.745613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffk9\" (UniqueName: \"kubernetes.io/projected/f8cac50e-c94f-4460-902c-5c2e7e3c075d-kube-api-access-nffk9\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:51.238549 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:51.238514    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-tls\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:51.241183 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:51.241154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8cac50e-c94f-4460-902c-5c2e7e3c075d-node-exporter-tls\") pod \"node-exporter-bgjwx\" (UID: \"f8cac50e-c94f-4460-902c-5c2e7e3c075d\") " pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:51.470165 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:51.470130    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgjwx"
Sep 16 12:49:51.849034 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:51.848999    2558 patch_prober.go:28] interesting pod/image-registry-84c8545869-j7bwr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:49:51.849424 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:51.849055    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84c8545869-j7bwr" podUID="1dd37e4c-4a3e-46f0-80d6-6c707af9e939" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:49:53.531258 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:53.531225    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:49:54.823763 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.823726    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"]
Sep 16 12:49:54.827944 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.827924    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:54.830731 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.830706    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 12:49:54.832003 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.831980    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 12:49:54.832116 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.832051    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1n5ap8mh0l0dv\""
Sep 16 12:49:54.832116 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.832064    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 12:49:54.832247 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.832192    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zb9mv\""
Sep 16 12:49:54.832315 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.832298    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:49:54.841956 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.841897    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"]
Sep 16 12:49:54.969751 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.969717    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e5f4771-0598-4865-9a90-299c9cbcd895-audit-log\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:54.969947 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.969768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-secret-metrics-server-client-certs\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:54.969947 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.969920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-secret-metrics-server-tls\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:54.970076 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.969971    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f4771-0598-4865-9a90-299c9cbcd895-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:54.970076 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.970048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk8dm\" (UniqueName: \"kubernetes.io/projected/7e5f4771-0598-4865-9a90-299c9cbcd895-kube-api-access-pk8dm\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:54.970076 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.970067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e5f4771-0598-4865-9a90-299c9cbcd895-metrics-server-audit-profiles\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:54.970200 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:54.970112    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-client-ca-bundle\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071081    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk8dm\" (UniqueName: \"kubernetes.io/projected/7e5f4771-0598-4865-9a90-299c9cbcd895-kube-api-access-pk8dm\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071136    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e5f4771-0598-4865-9a90-299c9cbcd895-metrics-server-audit-profiles\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071187    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-client-ca-bundle\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071213    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e5f4771-0598-4865-9a90-299c9cbcd895-audit-log\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071280 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-secret-metrics-server-client-certs\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071495 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-secret-metrics-server-tls\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071495 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f4771-0598-4865-9a90-299c9cbcd895-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.071780 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.071751    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e5f4771-0598-4865-9a90-299c9cbcd895-audit-log\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.072156 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.072101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f4771-0598-4865-9a90-299c9cbcd895-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.072339 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.072316    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e5f4771-0598-4865-9a90-299c9cbcd895-metrics-server-audit-profiles\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.074107 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.074045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-client-ca-bundle\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.074107 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.074056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-secret-metrics-server-tls\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.074246 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.074121    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e5f4771-0598-4865-9a90-299c9cbcd895-secret-metrics-server-client-certs\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.079761 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.079741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk8dm\" (UniqueName: \"kubernetes.io/projected/7e5f4771-0598-4865-9a90-299c9cbcd895-kube-api-access-pk8dm\") pod \"metrics-server-55b45f6cbd-7nzv8\" (UID: \"7e5f4771-0598-4865-9a90-299c9cbcd895\") " pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.138741 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.138703    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:49:55.310876 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.310844    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"]
Sep 16 12:49:55.314109 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.314072    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:55.319058 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.319035    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t4jq6\""
Sep 16 12:49:55.319164 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.319074    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 12:49:55.325310 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.325257    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"]
Sep 16 12:49:55.475400 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.475364    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pqtg\" (UniqueName: \"kubernetes.io/projected/4382446e-1349-4594-bf2e-e2633012b722-kube-api-access-6pqtg\") pod \"monitoring-plugin-5d85778b9d-hvcsv\" (UID: \"4382446e-1349-4594-bf2e-e2633012b722\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:55.475590 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.475423    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4382446e-1349-4594-bf2e-e2633012b722-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hvcsv\" (UID: \"4382446e-1349-4594-bf2e-e2633012b722\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:55.576742 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.576651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqtg\" (UniqueName: \"kubernetes.io/projected/4382446e-1349-4594-bf2e-e2633012b722-kube-api-access-6pqtg\") pod \"monitoring-plugin-5d85778b9d-hvcsv\" (UID: \"4382446e-1349-4594-bf2e-e2633012b722\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:55.576742 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.576700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4382446e-1349-4594-bf2e-e2633012b722-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hvcsv\" (UID: \"4382446e-1349-4594-bf2e-e2633012b722\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:55.576981 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:55.576895    2558 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 12:49:55.576981 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:49:55.576975    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4382446e-1349-4594-bf2e-e2633012b722-monitoring-plugin-cert podName:4382446e-1349-4594-bf2e-e2633012b722 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:56.076959689 +0000 UTC m=+122.538588363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4382446e-1349-4594-bf2e-e2633012b722-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-hvcsv" (UID: "4382446e-1349-4594-bf2e-e2633012b722") : secret "monitoring-plugin-cert" not found
Sep 16 12:49:55.592409 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:55.592380    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqtg\" (UniqueName: \"kubernetes.io/projected/4382446e-1349-4594-bf2e-e2633012b722-kube-api-access-6pqtg\") pod \"monitoring-plugin-5d85778b9d-hvcsv\" (UID: \"4382446e-1349-4594-bf2e-e2633012b722\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:56.080699 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:56.080657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4382446e-1349-4594-bf2e-e2633012b722-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hvcsv\" (UID: \"4382446e-1349-4594-bf2e-e2633012b722\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:56.083471 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:56.083446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4382446e-1349-4594-bf2e-e2633012b722-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hvcsv\" (UID: \"4382446e-1349-4594-bf2e-e2633012b722\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:56.224990 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:56.224952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:49:56.785655 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:56.785606    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cac50e_c94f_4460_902c_5c2e7e3c075d.slice/crio-4450ef71eef9a05b47555447909836b3fd3efa519f9d77ddc281adc1633057c4 WatchSource:0}: Error finding container 4450ef71eef9a05b47555447909836b3fd3efa519f9d77ddc281adc1633057c4: Status 404 returned error can't find the container with id 4450ef71eef9a05b47555447909836b3fd3efa519f9d77ddc281adc1633057c4
Sep 16 12:49:57.059014 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.058893    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"]
Sep 16 12:49:57.062703 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.062655    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"]
Sep 16 12:49:57.067954 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:57.067920    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f4771_0598_4865_9a90_299c9cbcd895.slice/crio-5cb8f92c522ebe1caec08e9e111a56f1a3d255995ef93a39d656cd8fdeeece58 WatchSource:0}: Error finding container 5cb8f92c522ebe1caec08e9e111a56f1a3d255995ef93a39d656cd8fdeeece58: Status 404 returned error can't find the container with id 5cb8f92c522ebe1caec08e9e111a56f1a3d255995ef93a39d656cd8fdeeece58
Sep 16 12:49:57.634871 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.634833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-c9hsx" event={"ID":"19f33b29-948e-4d9c-bb45-1e036c46358f","Type":"ContainerStarted","Data":"fe23bdaaed18cc7fdcbf7a91ba48d7f6ec4ab4dbc53983db98151b9924f93036"}
Sep 16 12:49:57.635425 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.635384    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-c9hsx"
Sep 16 12:49:57.637967 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.637933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgjwx" event={"ID":"f8cac50e-c94f-4460-902c-5c2e7e3c075d","Type":"ContainerStarted","Data":"4450ef71eef9a05b47555447909836b3fd3efa519f9d77ddc281adc1633057c4"}
Sep 16 12:49:57.640206 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.640172    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv" event={"ID":"4382446e-1349-4594-bf2e-e2633012b722","Type":"ContainerStarted","Data":"ae1023772f0145522c8918d0fc15c537707c27e9b96a0d0457bab04460ed7508"}
Sep 16 12:49:57.645689 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.645639    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8" event={"ID":"7e5f4771-0598-4865-9a90-299c9cbcd895","Type":"ContainerStarted","Data":"5cb8f92c522ebe1caec08e9e111a56f1a3d255995ef93a39d656cd8fdeeece58"}
Sep 16 12:49:57.655356 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.655333    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-c9hsx"
Sep 16 12:49:57.660012 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.658614    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-c9hsx" podStartSLOduration=2.428154382 podStartE2EDuration="19.658597926s" totalImagesPullingTime="17.230443544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:57.657256315 +0000 UTC m=+124.118885010" watchObservedRunningTime="2026-09-16 12:49:57.658597926 +0000 UTC m=+124.120226624"
Sep 16 12:49:57.968937 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:57.968362    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fc54b68f-4z55w"]
Sep 16 12:49:58.015544 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.015507    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7df7769976-pk6f9"]
Sep 16 12:49:58.026186 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.026154    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.031937 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.031913    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df7769976-pk6f9"]
Sep 16 12:49:58.199015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.198516    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-service-ca\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.199015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.198604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-serving-cert\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.199015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.198634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-console-config\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.199015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.198688    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-oauth-config\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.199015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.198713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp89x\" (UniqueName: \"kubernetes.io/projected/b0707266-6a8d-4756-95da-fe3657f63a1e-kube-api-access-fp89x\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.199015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.198741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-trusted-ca-bundle\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.199015 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.198788    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-oauth-serving-cert\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.299736 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.299702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-oauth-config\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.299936 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.299746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp89x\" (UniqueName: \"kubernetes.io/projected/b0707266-6a8d-4756-95da-fe3657f63a1e-kube-api-access-fp89x\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.299936 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.299773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-trusted-ca-bundle\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.300261 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.300147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-oauth-serving-cert\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.300261 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.300217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-service-ca\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.300872 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.300283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-serving-cert\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.300872 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.300312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-console-config\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.300990 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.300871    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-oauth-serving-cert\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.300990 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.300918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-console-config\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.301554 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.301525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-service-ca\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.301841 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.301736    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-trusted-ca-bundle\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.303895 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.303853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-oauth-config\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.304274 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.304239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-serving-cert\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.309278 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.309232    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp89x\" (UniqueName: \"kubernetes.io/projected/b0707266-6a8d-4756-95da-fe3657f63a1e-kube-api-access-fp89x\") pod \"console-7df7769976-pk6f9\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") " pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.340504 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.340458    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:49:58.431231 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.431183    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:49:58.652229 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:58.652187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgjwx" event={"ID":"f8cac50e-c94f-4460-902c-5c2e7e3c075d","Type":"ContainerStarted","Data":"5f7c20361db51dfd88c0f44701f0b507502035409bf5f3084af9d8cd88a51b69"}
Sep 16 12:49:59.658700 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:59.658670    2558 generic.go:356] "Generic (PLEG): container finished" podID="f8cac50e-c94f-4460-902c-5c2e7e3c075d" containerID="5f7c20361db51dfd88c0f44701f0b507502035409bf5f3084af9d8cd88a51b69" exitCode=0
Sep 16 12:49:59.659091 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:59.658752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgjwx" event={"ID":"f8cac50e-c94f-4460-902c-5c2e7e3c075d","Type":"ContainerDied","Data":"5f7c20361db51dfd88c0f44701f0b507502035409bf5f3084af9d8cd88a51b69"}
Sep 16 12:49:59.912529 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:49:59.912429    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df7769976-pk6f9"]
Sep 16 12:49:59.915466 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:49:59.915441    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0707266_6a8d_4756_95da_fe3657f63a1e.slice/crio-d49f9e0180ad79972073916675daecc853d9efb4e3a081ceb054ba7343e64a23 WatchSource:0}: Error finding container d49f9e0180ad79972073916675daecc853d9efb4e3a081ceb054ba7343e64a23: Status 404 returned error can't find the container with id d49f9e0180ad79972073916675daecc853d9efb4e3a081ceb054ba7343e64a23
Sep 16 12:50:00.665558 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.665515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgjwx" event={"ID":"f8cac50e-c94f-4460-902c-5c2e7e3c075d","Type":"ContainerStarted","Data":"8755c11cb72f5b2a45d51cbfd7b853d368b29392f14c64bcfe1693f4f7c01c0e"}
Sep 16 12:50:00.666047 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.665627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgjwx" event={"ID":"f8cac50e-c94f-4460-902c-5c2e7e3c075d","Type":"ContainerStarted","Data":"bfad71bfc27a3c693a301f92b2d6a57196a3f5eed7c5fa6f7a8cc3632f35ac6f"}
Sep 16 12:50:00.667310 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.667276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv" event={"ID":"4382446e-1349-4594-bf2e-e2633012b722","Type":"ContainerStarted","Data":"dc5251ad0b191ead899e72b5d714312ecf27892bdedce3c7cec82459599e8864"}
Sep 16 12:50:00.667504 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.667483    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:50:00.669142 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.669119    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8" event={"ID":"7e5f4771-0598-4865-9a90-299c9cbcd895","Type":"ContainerStarted","Data":"7c10560dfe1ff33ca047e71cb941132e7b7cc1ae391418756bb86b8e2c7ba5f1"}
Sep 16 12:50:00.671169 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.671148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df7769976-pk6f9" event={"ID":"b0707266-6a8d-4756-95da-fe3657f63a1e","Type":"ContainerStarted","Data":"a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1"}
Sep 16 12:50:00.671337 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.671317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df7769976-pk6f9" event={"ID":"b0707266-6a8d-4756-95da-fe3657f63a1e","Type":"ContainerStarted","Data":"d49f9e0180ad79972073916675daecc853d9efb4e3a081ceb054ba7343e64a23"}
Sep 16 12:50:00.673953 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.673935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv"
Sep 16 12:50:00.688517 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.688470    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bgjwx" podStartSLOduration=9.281207175 podStartE2EDuration="10.688456024s" totalImagesPullingTime="1.325767132s" totalInitContainerRuntime="81.481717ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:00.686913111 +0000 UTC m=+127.148541810" watchObservedRunningTime="2026-09-16 12:50:00.688456024 +0000 UTC m=+127.150084725"
Sep 16 12:50:00.704013 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.703972    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hvcsv" podStartSLOduration=3.139633072 podStartE2EDuration="5.703957422s" totalImagesPullingTime="2.56432435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:00.703693126 +0000 UTC m=+127.165321830" watchObservedRunningTime="2026-09-16 12:50:00.703957422 +0000 UTC m=+127.165586120"
Sep 16 12:50:00.723349 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.723298    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8" podStartSLOduration=4.156405557 podStartE2EDuration="6.723279076s" totalImagesPullingTime="2.566873519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:00.722777585 +0000 UTC m=+127.184406284" watchObservedRunningTime="2026-09-16 12:50:00.723279076 +0000 UTC m=+127.184907773"
Sep 16 12:50:00.742404 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:00.742352    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7df7769976-pk6f9" podStartSLOduration=3.742333783 podStartE2EDuration="3.742333783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:00.740794342 +0000 UTC m=+127.202423052" watchObservedRunningTime="2026-09-16 12:50:00.742333783 +0000 UTC m=+127.203962479"
Sep 16 12:50:01.960230 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:01.960192    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84c8545869-j7bwr"]
Sep 16 12:50:03.953307 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:03.953261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:50:03.956005 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:03.955972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d28d3da-9cae-48ec-ac36-003b8aa41bb6-metrics-certs\") pod \"network-metrics-daemon-cblhw\" (UID: \"9d28d3da-9cae-48ec-ac36-003b8aa41bb6\") " pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:50:04.051095 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:04.051062    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27x5j\""
Sep 16 12:50:04.058568 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:04.058538    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cblhw"
Sep 16 12:50:04.382979 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:04.382745    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cblhw"]
Sep 16 12:50:04.386034 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:50:04.385793    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d28d3da_9cae_48ec_ac36_003b8aa41bb6.slice/crio-bdcf4b4163883f89880c69becf71046798f9ce958df6fcd4f87fea04235dd73b WatchSource:0}: Error finding container bdcf4b4163883f89880c69becf71046798f9ce958df6fcd4f87fea04235dd73b: Status 404 returned error can't find the container with id bdcf4b4163883f89880c69becf71046798f9ce958df6fcd4f87fea04235dd73b
Sep 16 12:50:04.688759 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:04.688696    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cblhw" event={"ID":"9d28d3da-9cae-48ec-ac36-003b8aa41bb6","Type":"ContainerStarted","Data":"bdcf4b4163883f89880c69becf71046798f9ce958df6fcd4f87fea04235dd73b"}
Sep 16 12:50:06.697664 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:06.697625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cblhw" event={"ID":"9d28d3da-9cae-48ec-ac36-003b8aa41bb6","Type":"ContainerStarted","Data":"7df3f15ada0355d1f6e7711833c579962f45b49269738192d232514da855353a"}
Sep 16 12:50:06.698245 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:06.697685    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cblhw" event={"ID":"9d28d3da-9cae-48ec-ac36-003b8aa41bb6","Type":"ContainerStarted","Data":"f934f4dca35d7f772e5f897ac67bc0b36a65fdbe95bf63ddc4c529c4d51832e0"}
Sep 16 12:50:06.717264 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:06.717210    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cblhw" podStartSLOduration=131.329736662 podStartE2EDuration="2m12.717192806s" totalImagesPullingTime="1.387456144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:06.715879467 +0000 UTC m=+133.177508165" watchObservedRunningTime="2026-09-16 12:50:06.717192806 +0000 UTC m=+133.178821503"
Sep 16 12:50:08.341614 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:08.341465    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:50:08.342089 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:08.341702    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:50:08.347239 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:08.347216    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:50:08.710671 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:08.710639    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:50:08.760136 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:08.760102    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6676594c-rvb4z"]
Sep 16 12:50:15.139358 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:15.139327    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:50:15.139853 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:15.139384    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:50:22.997619 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:22.997552    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65fc54b68f-4z55w" podUID="7581b5ab-919c-4ad2-9141-9f76f2da17f7" containerName="console" containerID="cri-o://df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da" gracePeriod=15
Sep 16 12:50:23.229875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.229851    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:50:23.302962 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.302896    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-oauth-serving-cert\") pod \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") "
Sep 16 12:50:23.302962 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.302926    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-service-ca\") pod \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") "
Sep 16 12:50:23.303138 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.302968    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-serving-cert\") pod \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") "
Sep 16 12:50:23.303138 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303031    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-config\") pod \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") "
Sep 16 12:50:23.303138 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303056    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm7ph\" (UniqueName: \"kubernetes.io/projected/7581b5ab-919c-4ad2-9141-9f76f2da17f7-kube-api-access-pm7ph\") pod \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") "
Sep 16 12:50:23.303138 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303100    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-oauth-config\") pod \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") "
Sep 16 12:50:23.303138 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303123    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-trusted-ca-bundle\") pod \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\" (UID: \"7581b5ab-919c-4ad2-9141-9f76f2da17f7\") "
Sep 16 12:50:23.303376 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303311    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7581b5ab-919c-4ad2-9141-9f76f2da17f7" (UID: "7581b5ab-919c-4ad2-9141-9f76f2da17f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:23.303502 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303472    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "7581b5ab-919c-4ad2-9141-9f76f2da17f7" (UID: "7581b5ab-919c-4ad2-9141-9f76f2da17f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:23.303576 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303551    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-config" (OuterVolumeSpecName: "console-config") pod "7581b5ab-919c-4ad2-9141-9f76f2da17f7" (UID: "7581b5ab-919c-4ad2-9141-9f76f2da17f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:23.303652 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.303625    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7581b5ab-919c-4ad2-9141-9f76f2da17f7" (UID: "7581b5ab-919c-4ad2-9141-9f76f2da17f7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:23.305385 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.305362    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7581b5ab-919c-4ad2-9141-9f76f2da17f7" (UID: "7581b5ab-919c-4ad2-9141-9f76f2da17f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:23.305461 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.305385    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7581b5ab-919c-4ad2-9141-9f76f2da17f7" (UID: "7581b5ab-919c-4ad2-9141-9f76f2da17f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:23.305461 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.305429    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7581b5ab-919c-4ad2-9141-9f76f2da17f7-kube-api-access-pm7ph" (OuterVolumeSpecName: "kube-api-access-pm7ph") pod "7581b5ab-919c-4ad2-9141-9f76f2da17f7" (UID: "7581b5ab-919c-4ad2-9141-9f76f2da17f7"). InnerVolumeSpecName "kube-api-access-pm7ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:23.404149 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.404117    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.404149 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.404145    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7ph\" (UniqueName: \"kubernetes.io/projected/7581b5ab-919c-4ad2-9141-9f76f2da17f7-kube-api-access-pm7ph\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.404149 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.404156    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-oauth-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.404354 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.404165    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-trusted-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.404354 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.404173    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-oauth-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.404354 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.404182    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7581b5ab-919c-4ad2-9141-9f76f2da17f7-service-ca\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.404354 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.404192    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7581b5ab-919c-4ad2-9141-9f76f2da17f7-console-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.764467 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.764432    2558 generic.go:356] "Generic (PLEG): container finished" podID="7581b5ab-919c-4ad2-9141-9f76f2da17f7" containerID="df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da" exitCode=0
Sep 16 12:50:23.764467 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.764470    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fc54b68f-4z55w"
Sep 16 12:50:23.764672 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.764494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fc54b68f-4z55w" event={"ID":"7581b5ab-919c-4ad2-9141-9f76f2da17f7","Type":"ContainerDied","Data":"df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da"}
Sep 16 12:50:23.764672 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.764542    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fc54b68f-4z55w" event={"ID":"7581b5ab-919c-4ad2-9141-9f76f2da17f7","Type":"ContainerDied","Data":"caa70fbec3fd4dbb664e8ebed01051422e85c3a6da863b61cd5dd6a37c10d71a"}
Sep 16 12:50:23.764672 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.764565    2558 scope.go:117] "RemoveContainer" containerID="df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da"
Sep 16 12:50:23.773210 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.773191    2558 scope.go:117] "RemoveContainer" containerID="df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da"
Sep 16 12:50:23.773498 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:50:23.773477    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da\": container with ID starting with df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da not found: ID does not exist" containerID="df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da"
Sep 16 12:50:23.773573 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.773510    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da"} err="failed to get container status \"df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da\": rpc error: code = NotFound desc = could not find container \"df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da\": container with ID starting with df9366356c30a074013aa69186a0ec1cf609995f8437f88663937268388b50da not found: ID does not exist"
Sep 16 12:50:23.786616 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.786593    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fc54b68f-4z55w"]
Sep 16 12:50:23.789980 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:23.789946    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65fc54b68f-4z55w"]
Sep 16 12:50:24.041527 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:24.041437    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7581b5ab-919c-4ad2-9141-9f76f2da17f7" path="/var/lib/kubelet/pods/7581b5ab-919c-4ad2-9141-9f76f2da17f7/volumes"
Sep 16 12:50:26.985089 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:26.985034    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84c8545869-j7bwr" podUID="1dd37e4c-4a3e-46f0-80d6-6c707af9e939" containerName="registry" containerID="cri-o://bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841" gracePeriod=30
Sep 16 12:50:27.271319 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.271288    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:50:27.334121 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjq5r\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-kube-api-access-sjq5r\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.334301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334147    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-image-registry-private-configuration\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.334301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334173    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-installation-pull-secrets\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.334301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334218    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-certificates\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.334301 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334243    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-trusted-ca\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.334532 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334384    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-ca-trust-extracted\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.334532 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334468    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-bound-sa-token\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.334532 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.334521    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") pod \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\" (UID: \"1dd37e4c-4a3e-46f0-80d6-6c707af9e939\") "
Sep 16 12:50:27.335253 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.335129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:27.336291 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.335647    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:27.338195 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.337019    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:27.341389 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.338565    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:27.341717 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.341680    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-kube-api-access-sjq5r" (OuterVolumeSpecName: "kube-api-access-sjq5r") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "kube-api-access-sjq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:27.341841 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.341716    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:27.341841 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.341803    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:27.347568 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.347545    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1dd37e4c-4a3e-46f0-80d6-6c707af9e939" (UID: "1dd37e4c-4a3e-46f0-80d6-6c707af9e939"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:50:27.436198 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436161    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.436198 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436192    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjq5r\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-kube-api-access-sjq5r\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.436198 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436204    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-image-registry-private-configuration\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.436418 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436213    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-installation-pull-secrets\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.436418 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436222    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-registry-certificates\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.436418 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436230    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-trusted-ca\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.436418 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436238    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-ca-trust-extracted\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.436418 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.436247    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dd37e4c-4a3e-46f0-80d6-6c707af9e939-bound-sa-token\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:27.780462 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.780423    2558 generic.go:356] "Generic (PLEG): container finished" podID="1dd37e4c-4a3e-46f0-80d6-6c707af9e939" containerID="bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841" exitCode=0
Sep 16 12:50:27.780658 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.780490    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c8545869-j7bwr"
Sep 16 12:50:27.780658 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.780511    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c8545869-j7bwr" event={"ID":"1dd37e4c-4a3e-46f0-80d6-6c707af9e939","Type":"ContainerDied","Data":"bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841"}
Sep 16 12:50:27.780658 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.780605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c8545869-j7bwr" event={"ID":"1dd37e4c-4a3e-46f0-80d6-6c707af9e939","Type":"ContainerDied","Data":"4ca9bf6d20beee8ef8429b499f79ac66d82eff90119548be73fa5f65902b2918"}
Sep 16 12:50:27.780658 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.780634    2558 scope.go:117] "RemoveContainer" containerID="bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841"
Sep 16 12:50:27.788398 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.788378    2558 scope.go:117] "RemoveContainer" containerID="bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841"
Sep 16 12:50:27.788663 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:50:27.788640    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841\": container with ID starting with bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841 not found: ID does not exist" containerID="bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841"
Sep 16 12:50:27.788739 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.788670    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841"} err="failed to get container status \"bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841\": rpc error: code = NotFound desc = could not find container \"bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841\": container with ID starting with bd1a263366153969de2298d6527b6d4f6e61367b972cf20eded1e09bd676b841 not found: ID does not exist"
Sep 16 12:50:27.802664 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.802642    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84c8545869-j7bwr"]
Sep 16 12:50:27.814154 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:27.814136    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84c8545869-j7bwr"]
Sep 16 12:50:28.041870 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:28.041781    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd37e4c-4a3e-46f0-80d6-6c707af9e939" path="/var/lib/kubelet/pods/1dd37e4c-4a3e-46f0-80d6-6c707af9e939/volumes"
Sep 16 12:50:33.783646 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:33.783587    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b6676594c-rvb4z" podUID="51a4d013-d7de-4582-aeca-74b2accfe812" containerName="console" containerID="cri-o://cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22" gracePeriod=15
Sep 16 12:50:34.064241 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.064178    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:50:34.183875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.183838    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-oauth-serving-cert\") pod \"51a4d013-d7de-4582-aeca-74b2accfe812\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") "
Sep 16 12:50:34.184052 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.183885    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-serving-cert\") pod \"51a4d013-d7de-4582-aeca-74b2accfe812\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") "
Sep 16 12:50:34.184052 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.183940    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-oauth-config\") pod \"51a4d013-d7de-4582-aeca-74b2accfe812\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") "
Sep 16 12:50:34.184052 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.183967    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-service-ca\") pod \"51a4d013-d7de-4582-aeca-74b2accfe812\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") "
Sep 16 12:50:34.184052 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.184001    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvdh7\" (UniqueName: \"kubernetes.io/projected/51a4d013-d7de-4582-aeca-74b2accfe812-kube-api-access-mvdh7\") pod \"51a4d013-d7de-4582-aeca-74b2accfe812\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") "
Sep 16 12:50:34.184052 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.184022    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-console-config\") pod \"51a4d013-d7de-4582-aeca-74b2accfe812\" (UID: \"51a4d013-d7de-4582-aeca-74b2accfe812\") "
Sep 16 12:50:34.184322 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.184289    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "51a4d013-d7de-4582-aeca-74b2accfe812" (UID: "51a4d013-d7de-4582-aeca-74b2accfe812"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:34.184434 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.184400    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-service-ca" (OuterVolumeSpecName: "service-ca") pod "51a4d013-d7de-4582-aeca-74b2accfe812" (UID: "51a4d013-d7de-4582-aeca-74b2accfe812"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:34.184434 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.184411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-console-config" (OuterVolumeSpecName: "console-config") pod "51a4d013-d7de-4582-aeca-74b2accfe812" (UID: "51a4d013-d7de-4582-aeca-74b2accfe812"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:34.186196 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.186175    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a4d013-d7de-4582-aeca-74b2accfe812-kube-api-access-mvdh7" (OuterVolumeSpecName: "kube-api-access-mvdh7") pod "51a4d013-d7de-4582-aeca-74b2accfe812" (UID: "51a4d013-d7de-4582-aeca-74b2accfe812"). InnerVolumeSpecName "kube-api-access-mvdh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:34.186196 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.186180    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "51a4d013-d7de-4582-aeca-74b2accfe812" (UID: "51a4d013-d7de-4582-aeca-74b2accfe812"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:34.186325 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.186236    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "51a4d013-d7de-4582-aeca-74b2accfe812" (UID: "51a4d013-d7de-4582-aeca-74b2accfe812"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:34.285511 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.285467    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-oauth-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:34.285511 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.285509    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-service-ca\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:34.285511 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.285521    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvdh7\" (UniqueName: \"kubernetes.io/projected/51a4d013-d7de-4582-aeca-74b2accfe812-kube-api-access-mvdh7\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:34.285511 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.285529    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-console-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:34.285511 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.285538    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a4d013-d7de-4582-aeca-74b2accfe812-oauth-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:34.285838 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.285546    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a4d013-d7de-4582-aeca-74b2accfe812-console-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:50:34.806851 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.806803    2558 generic.go:356] "Generic (PLEG): container finished" podID="51a4d013-d7de-4582-aeca-74b2accfe812" containerID="cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22" exitCode=0
Sep 16 12:50:34.806851 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.806853    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6676594c-rvb4z"
Sep 16 12:50:34.807317 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.806888    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6676594c-rvb4z" event={"ID":"51a4d013-d7de-4582-aeca-74b2accfe812","Type":"ContainerDied","Data":"cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22"}
Sep 16 12:50:34.807317 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.806925    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6676594c-rvb4z" event={"ID":"51a4d013-d7de-4582-aeca-74b2accfe812","Type":"ContainerDied","Data":"1ab90abd8742a98f81e45519add5800c63a4fa88208bd5aa49b1a737505033a3"}
Sep 16 12:50:34.807317 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.806945    2558 scope.go:117] "RemoveContainer" containerID="cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22"
Sep 16 12:50:34.814799 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.814781    2558 scope.go:117] "RemoveContainer" containerID="cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22"
Sep 16 12:50:34.815079 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:50:34.815059    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22\": container with ID starting with cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22 not found: ID does not exist" containerID="cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22"
Sep 16 12:50:34.815149 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.815087    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22"} err="failed to get container status \"cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22\": rpc error: code = NotFound desc = could not find container \"cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22\": container with ID starting with cd361b30a5afa88ac3ff3c7e5a581eca9a1538565e4ba9a711f80ca3c2b65e22 not found: ID does not exist"
Sep 16 12:50:34.827970 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.827946    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6676594c-rvb4z"]
Sep 16 12:50:34.831144 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:34.831122    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b6676594c-rvb4z"]
Sep 16 12:50:35.144605 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:35.144578    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:50:35.148473 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:35.148452    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55b45f6cbd-7nzv8"
Sep 16 12:50:36.041173 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:50:36.041137    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a4d013-d7de-4582-aeca-74b2accfe812" path="/var/lib/kubelet/pods/51a4d013-d7de-4582-aeca-74b2accfe812/volumes"
Sep 16 12:51:18.461268 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461238    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b888b5d4b-tsdx6"]
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461477    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7581b5ab-919c-4ad2-9141-9f76f2da17f7" containerName="console"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461491    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7581b5ab-919c-4ad2-9141-9f76f2da17f7" containerName="console"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461507    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a4d013-d7de-4582-aeca-74b2accfe812" containerName="console"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461513    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a4d013-d7de-4582-aeca-74b2accfe812" containerName="console"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461524    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd37e4c-4a3e-46f0-80d6-6c707af9e939" containerName="registry"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461529    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd37e4c-4a3e-46f0-80d6-6c707af9e939" containerName="registry"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461580    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd37e4c-4a3e-46f0-80d6-6c707af9e939" containerName="registry"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461588    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a4d013-d7de-4582-aeca-74b2accfe812" containerName="console"
Sep 16 12:51:18.461687 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.461595    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7581b5ab-919c-4ad2-9141-9f76f2da17f7" containerName="console"
Sep 16 12:51:18.464337 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.464316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.480901 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.480873    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b888b5d4b-tsdx6"]
Sep 16 12:51:18.613062 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.613023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-trusted-ca-bundle\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.613062 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.613060    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-service-ca\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.613267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.613077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5h9\" (UniqueName: \"kubernetes.io/projected/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-kube-api-access-pk5h9\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.613267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.613145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-config\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.613267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.613182    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-oauth-config\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.613267 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.613223    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-serving-cert\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.613393 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.613269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-oauth-serving-cert\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.713652 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.713563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-oauth-serving-cert\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.713652 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.713614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-trusted-ca-bundle\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.713652 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.713637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-service-ca\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.713963 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.713660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5h9\" (UniqueName: \"kubernetes.io/projected/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-kube-api-access-pk5h9\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.713963 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.713705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-config\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.713963 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.713728    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-oauth-config\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.713963 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.713758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-serving-cert\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.714444 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.714416    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-oauth-serving-cert\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.714576 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.714492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-config\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.714640 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.714492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-service-ca\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.714640 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.714510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-trusted-ca-bundle\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.716116 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.716097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-serving-cert\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.716201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.716182    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-oauth-config\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.722642 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.722621    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5h9\" (UniqueName: \"kubernetes.io/projected/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-kube-api-access-pk5h9\") pod \"console-b888b5d4b-tsdx6\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") " pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:18.773084 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:18.773060    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:19.049604 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:19.049572    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b888b5d4b-tsdx6"]
Sep 16 12:51:19.052425 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:51:19.052392    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff78d04_cfa8_46e7_8d83_e7a7b4da64ae.slice/crio-9bc5535d81b80d16dbf55b0c817219270f1457693c91ba8249e93312a9a8f67b WatchSource:0}: Error finding container 9bc5535d81b80d16dbf55b0c817219270f1457693c91ba8249e93312a9a8f67b: Status 404 returned error can't find the container with id 9bc5535d81b80d16dbf55b0c817219270f1457693c91ba8249e93312a9a8f67b
Sep 16 12:51:19.968470 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:19.968436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b888b5d4b-tsdx6" event={"ID":"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae","Type":"ContainerStarted","Data":"1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446"}
Sep 16 12:51:19.968871 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:19.968484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b888b5d4b-tsdx6" event={"ID":"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae","Type":"ContainerStarted","Data":"9bc5535d81b80d16dbf55b0c817219270f1457693c91ba8249e93312a9a8f67b"}
Sep 16 12:51:19.986715 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:19.986666    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b888b5d4b-tsdx6" podStartSLOduration=1.986650177 podStartE2EDuration="1.986650177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:19.985249171 +0000 UTC m=+206.446877867" watchObservedRunningTime="2026-09-16 12:51:19.986650177 +0000 UTC m=+206.448278872"
Sep 16 12:51:28.773760 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:28.773721    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:28.774306 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:28.773782    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:28.778482 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:28.778458    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:29.002454 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.002427    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:51:29.052455 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.052368    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df7769976-pk6f9"]
Sep 16 12:51:29.161569 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.161538    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77f7896f69-867wg"]
Sep 16 12:51:29.164872 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.164855    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.174590 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.174567    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f7896f69-867wg"]
Sep 16 12:51:29.298474 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.298435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-serving-cert\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.298474 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.298477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-config\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.298706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.298536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmzc\" (UniqueName: \"kubernetes.io/projected/8ac9a191-da2a-4774-86fa-9f29a9659e38-kube-api-access-gbmzc\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.298706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.298586    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-oauth-serving-cert\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.298706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.298605    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-service-ca\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.298706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.298626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-oauth-config\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.298706 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.298649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-trusted-ca-bundle\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399039 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-oauth-serving-cert\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-service-ca\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-oauth-config\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-trusted-ca-bundle\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-serving-cert\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399201 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-config\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399388 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399219    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmzc\" (UniqueName: \"kubernetes.io/projected/8ac9a191-da2a-4774-86fa-9f29a9659e38-kube-api-access-gbmzc\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399849 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-service-ca\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.399985 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.399946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-oauth-serving-cert\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.400048 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.400016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-trusted-ca-bundle\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.400227 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.400199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-config\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.401860 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.401833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-serving-cert\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.401948 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.401833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-oauth-config\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.407771 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.407752    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmzc\" (UniqueName: \"kubernetes.io/projected/8ac9a191-da2a-4774-86fa-9f29a9659e38-kube-api-access-gbmzc\") pod \"console-77f7896f69-867wg\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") " pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.474477 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.474445    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:29.745878 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:29.745770    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f7896f69-867wg"]
Sep 16 12:51:29.748606 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:51:29.748576    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac9a191_da2a_4774_86fa_9f29a9659e38.slice/crio-3b3c5514b4cf3f2624d39d96cdde25d149558e52c9e27e38abce11f81d004a85 WatchSource:0}: Error finding container 3b3c5514b4cf3f2624d39d96cdde25d149558e52c9e27e38abce11f81d004a85: Status 404 returned error can't find the container with id 3b3c5514b4cf3f2624d39d96cdde25d149558e52c9e27e38abce11f81d004a85
Sep 16 12:51:30.003402 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:30.003306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f7896f69-867wg" event={"ID":"8ac9a191-da2a-4774-86fa-9f29a9659e38","Type":"ContainerStarted","Data":"d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381"}
Sep 16 12:51:30.003402 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:30.003363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f7896f69-867wg" event={"ID":"8ac9a191-da2a-4774-86fa-9f29a9659e38","Type":"ContainerStarted","Data":"3b3c5514b4cf3f2624d39d96cdde25d149558e52c9e27e38abce11f81d004a85"}
Sep 16 12:51:39.475490 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:39.475442    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:39.475994 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:39.475509    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:39.480327 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:39.480305    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:39.501766 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:39.498388    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77f7896f69-867wg" podStartSLOduration=10.498369437000001 podStartE2EDuration="10.498369437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:30.028972994 +0000 UTC m=+216.490601689" watchObservedRunningTime="2026-09-16 12:51:39.498369437 +0000 UTC m=+225.959998133"
Sep 16 12:51:40.042620 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:40.042591    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:51:40.094975 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:40.094942    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b888b5d4b-tsdx6"]
Sep 16 12:51:54.076500 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.076459    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7df7769976-pk6f9" podUID="b0707266-6a8d-4756-95da-fe3657f63a1e" containerName="console" containerID="cri-o://a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1" gracePeriod=15
Sep 16 12:51:54.312932 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.312907    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:51:54.385117 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385045    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-oauth-config\") pod \"b0707266-6a8d-4756-95da-fe3657f63a1e\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") "
Sep 16 12:51:54.385117 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385080    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp89x\" (UniqueName: \"kubernetes.io/projected/b0707266-6a8d-4756-95da-fe3657f63a1e-kube-api-access-fp89x\") pod \"b0707266-6a8d-4756-95da-fe3657f63a1e\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") "
Sep 16 12:51:54.385279 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385118    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-service-ca\") pod \"b0707266-6a8d-4756-95da-fe3657f63a1e\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") "
Sep 16 12:51:54.385279 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385163    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-oauth-serving-cert\") pod \"b0707266-6a8d-4756-95da-fe3657f63a1e\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") "
Sep 16 12:51:54.385279 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385206    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-serving-cert\") pod \"b0707266-6a8d-4756-95da-fe3657f63a1e\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") "
Sep 16 12:51:54.385279 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385264    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-console-config\") pod \"b0707266-6a8d-4756-95da-fe3657f63a1e\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") "
Sep 16 12:51:54.385467 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385288    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-trusted-ca-bundle\") pod \"b0707266-6a8d-4756-95da-fe3657f63a1e\" (UID: \"b0707266-6a8d-4756-95da-fe3657f63a1e\") "
Sep 16 12:51:54.385713 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385633    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-service-ca" (OuterVolumeSpecName: "service-ca") pod "b0707266-6a8d-4756-95da-fe3657f63a1e" (UID: "b0707266-6a8d-4756-95da-fe3657f63a1e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:54.385806 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385690    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b0707266-6a8d-4756-95da-fe3657f63a1e" (UID: "b0707266-6a8d-4756-95da-fe3657f63a1e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:54.385806 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385728    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-console-config" (OuterVolumeSpecName: "console-config") pod "b0707266-6a8d-4756-95da-fe3657f63a1e" (UID: "b0707266-6a8d-4756-95da-fe3657f63a1e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:54.385916 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.385855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b0707266-6a8d-4756-95da-fe3657f63a1e" (UID: "b0707266-6a8d-4756-95da-fe3657f63a1e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:54.387155 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.387133    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b0707266-6a8d-4756-95da-fe3657f63a1e" (UID: "b0707266-6a8d-4756-95da-fe3657f63a1e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:54.387577 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.387548    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b0707266-6a8d-4756-95da-fe3657f63a1e" (UID: "b0707266-6a8d-4756-95da-fe3657f63a1e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:54.387577 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.387562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0707266-6a8d-4756-95da-fe3657f63a1e-kube-api-access-fp89x" (OuterVolumeSpecName: "kube-api-access-fp89x") pod "b0707266-6a8d-4756-95da-fe3657f63a1e" (UID: "b0707266-6a8d-4756-95da-fe3657f63a1e"). InnerVolumeSpecName "kube-api-access-fp89x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:51:54.486161 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.486111    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-oauth-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:51:54.486161 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.486155    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp89x\" (UniqueName: \"kubernetes.io/projected/b0707266-6a8d-4756-95da-fe3657f63a1e-kube-api-access-fp89x\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:51:54.486161 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.486167    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-service-ca\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:51:54.486161 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.486175    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-oauth-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:51:54.486416 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.486183    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0707266-6a8d-4756-95da-fe3657f63a1e-console-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:51:54.486416 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.486191    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-console-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:51:54.486416 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:54.486199    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0707266-6a8d-4756-95da-fe3657f63a1e-trusted-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:51:55.087672 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.087641    2558 generic.go:356] "Generic (PLEG): container finished" podID="b0707266-6a8d-4756-95da-fe3657f63a1e" containerID="a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1" exitCode=0
Sep 16 12:51:55.088122 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.087688    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df7769976-pk6f9"
Sep 16 12:51:55.088122 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.087697    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df7769976-pk6f9" event={"ID":"b0707266-6a8d-4756-95da-fe3657f63a1e","Type":"ContainerDied","Data":"a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1"}
Sep 16 12:51:55.088122 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.087731    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df7769976-pk6f9" event={"ID":"b0707266-6a8d-4756-95da-fe3657f63a1e","Type":"ContainerDied","Data":"d49f9e0180ad79972073916675daecc853d9efb4e3a081ceb054ba7343e64a23"}
Sep 16 12:51:55.088122 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.087754    2558 scope.go:117] "RemoveContainer" containerID="a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1"
Sep 16 12:51:55.096146 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.096129    2558 scope.go:117] "RemoveContainer" containerID="a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1"
Sep 16 12:51:55.096404 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:51:55.096387    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1\": container with ID starting with a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1 not found: ID does not exist" containerID="a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1"
Sep 16 12:51:55.096449 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.096414    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1"} err="failed to get container status \"a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1\": rpc error: code = NotFound desc = could not find container \"a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1\": container with ID starting with a872ff23c4307715394b7a8eb4e717de2286e400fc5edd38e6ee11c46d01e5e1 not found: ID does not exist"
Sep 16 12:51:55.108542 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.108519    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df7769976-pk6f9"]
Sep 16 12:51:55.112362 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:55.112342    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7df7769976-pk6f9"]
Sep 16 12:51:56.041145 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:51:56.041116    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0707266-6a8d-4756-95da-fe3657f63a1e" path="/var/lib/kubelet/pods/b0707266-6a8d-4756-95da-fe3657f63a1e/volumes"
Sep 16 12:52:05.114647 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.114594    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b888b5d4b-tsdx6" podUID="fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" containerName="console" containerID="cri-o://1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446" gracePeriod=15
Sep 16 12:52:05.347785 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.347763    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:52:05.464409 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464374    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-oauth-config\") pod \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") "
Sep 16 12:52:05.464586 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464420    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-config\") pod \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") "
Sep 16 12:52:05.464586 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464443    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-oauth-serving-cert\") pod \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") "
Sep 16 12:52:05.464586 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464472    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-trusted-ca-bundle\") pod \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") "
Sep 16 12:52:05.464586 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464509    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk5h9\" (UniqueName: \"kubernetes.io/projected/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-kube-api-access-pk5h9\") pod \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") "
Sep 16 12:52:05.464586 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464544    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-serving-cert\") pod \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") "
Sep 16 12:52:05.464857 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464633    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-service-ca\") pod \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\" (UID: \"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae\") "
Sep 16 12:52:05.464933 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464907    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" (UID: "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:05.464997 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.464919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-config" (OuterVolumeSpecName: "console-config") pod "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" (UID: "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:05.465112 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.465089    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" (UID: "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:05.465169 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.465151    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-service-ca" (OuterVolumeSpecName: "service-ca") pod "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" (UID: "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:05.466749 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.466727    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-kube-api-access-pk5h9" (OuterVolumeSpecName: "kube-api-access-pk5h9") pod "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" (UID: "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae"). InnerVolumeSpecName "kube-api-access-pk5h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:52:05.466865 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.466730    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" (UID: "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:52:05.466865 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.466830    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" (UID: "fff78d04-cfa8-46e7-8d83-e7a7b4da64ae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:52:05.565437 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.565402    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-oauth-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:52:05.565437 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.565431    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:52:05.565437 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.565440    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-oauth-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:52:05.565647 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.565449    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-trusted-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:52:05.565647 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.565458    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk5h9\" (UniqueName: \"kubernetes.io/projected/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-kube-api-access-pk5h9\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:52:05.565647 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.565470    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-console-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:52:05.565647 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:05.565481    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae-service-ca\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:52:06.126877 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.126845    2558 generic.go:356] "Generic (PLEG): container finished" podID="fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" containerID="1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446" exitCode=0
Sep 16 12:52:06.127268 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.126897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b888b5d4b-tsdx6" event={"ID":"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae","Type":"ContainerDied","Data":"1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446"}
Sep 16 12:52:06.127268 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.126942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b888b5d4b-tsdx6" event={"ID":"fff78d04-cfa8-46e7-8d83-e7a7b4da64ae","Type":"ContainerDied","Data":"9bc5535d81b80d16dbf55b0c817219270f1457693c91ba8249e93312a9a8f67b"}
Sep 16 12:52:06.127268 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.126942    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b888b5d4b-tsdx6"
Sep 16 12:52:06.127268 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.126963    2558 scope.go:117] "RemoveContainer" containerID="1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446"
Sep 16 12:52:06.134574 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.134550    2558 scope.go:117] "RemoveContainer" containerID="1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446"
Sep 16 12:52:06.134858 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:52:06.134839    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446\": container with ID starting with 1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446 not found: ID does not exist" containerID="1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446"
Sep 16 12:52:06.134927 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.134866    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446"} err="failed to get container status \"1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446\": rpc error: code = NotFound desc = could not find container \"1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446\": container with ID starting with 1ee110dab26d681d7adbe59e2ef92553480509a54a9ebd780763df3806478446 not found: ID does not exist"
Sep 16 12:52:06.144835 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.144798    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b888b5d4b-tsdx6"]
Sep 16 12:52:06.146479 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:06.146460    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b888b5d4b-tsdx6"]
Sep 16 12:52:08.041405 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:08.041368    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" path="/var/lib/kubelet/pods/fff78d04-cfa8-46e7-8d83-e7a7b4da64ae/volumes"
Sep 16 12:52:12.218217 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.218183    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"]
Sep 16 12:52:12.218665 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.218415    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" containerName="console"
Sep 16 12:52:12.218665 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.218425    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" containerName="console"
Sep 16 12:52:12.218665 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.218438    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0707266-6a8d-4756-95da-fe3657f63a1e" containerName="console"
Sep 16 12:52:12.218665 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.218443    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0707266-6a8d-4756-95da-fe3657f63a1e" containerName="console"
Sep 16 12:52:12.218665 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.218479    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff78d04-cfa8-46e7-8d83-e7a7b4da64ae" containerName="console"
Sep 16 12:52:12.218665 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.218488    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0707266-6a8d-4756-95da-fe3657f63a1e" containerName="console"
Sep 16 12:52:12.222707 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.222686    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.226548 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.226525    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:52:12.226548 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.226543    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:52:12.226703 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.226528    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:52:12.226703 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.226525    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 12:52:12.232411 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.232393    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"]
Sep 16 12:52:12.258741 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.258718    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"]
Sep 16 12:52:12.261782 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.261767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.264330 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.264300    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 12:52:12.264430 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.264387    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 12:52:12.264612 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.264595    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 12:52:12.264711 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.264695    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 12:52:12.273251 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.273228    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"]
Sep 16 12:52:12.310735 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6hl\" (UniqueName: \"kubernetes.io/projected/5b27d374-cb23-4b62-867d-c4a34fbb235e-kube-api-access-vd6hl\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.310875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5b27d374-cb23-4b62-867d-c4a34fbb235e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.310875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-hub\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.310875 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310795    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ngt8\" (UniqueName: \"kubernetes.io/projected/773b717d-e15b-4aa9-a9fc-a61950b67915-kube-api-access-5ngt8\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.311000 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.311000 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310942    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b27d374-cb23-4b62-867d-c4a34fbb235e-tmp\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.311000 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-ca\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.311000 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.310978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.311131 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.311010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/773b717d-e15b-4aa9-a9fc-a61950b67915-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.411581 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-ca\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.411756 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.411756 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/773b717d-e15b-4aa9-a9fc-a61950b67915-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.411756 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6hl\" (UniqueName: \"kubernetes.io/projected/5b27d374-cb23-4b62-867d-c4a34fbb235e-kube-api-access-vd6hl\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.411756 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5b27d374-cb23-4b62-867d-c4a34fbb235e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.411756 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-hub\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.411756 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngt8\" (UniqueName: \"kubernetes.io/projected/773b717d-e15b-4aa9-a9fc-a61950b67915-kube-api-access-5ngt8\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.412112 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411800    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.412112 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.411852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b27d374-cb23-4b62-867d-c4a34fbb235e-tmp\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.412293 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.412271    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b27d374-cb23-4b62-867d-c4a34fbb235e-tmp\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.412490 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.412466    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/773b717d-e15b-4aa9-a9fc-a61950b67915-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.414390 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.414366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-hub\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.414521 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.414505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-ca\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.414607 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.414587    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.414756 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.414732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5b27d374-cb23-4b62-867d-c4a34fbb235e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.414850 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.414837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/773b717d-e15b-4aa9-a9fc-a61950b67915-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.419966 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.419940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6hl\" (UniqueName: \"kubernetes.io/projected/5b27d374-cb23-4b62-867d-c4a34fbb235e-kube-api-access-vd6hl\") pod \"klusterlet-addon-workmgr-6d46c9cfbc-66wbc\" (UID: \"5b27d374-cb23-4b62-867d-c4a34fbb235e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.420137 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.420118    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ngt8\" (UniqueName: \"kubernetes.io/projected/773b717d-e15b-4aa9-a9fc-a61950b67915-kube-api-access-5ngt8\") pod \"cluster-proxy-proxy-agent-6c6c658485-wrr2s\" (UID: \"773b717d-e15b-4aa9-a9fc-a61950b67915\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.531779 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.531682    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:12.593273 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.593217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"
Sep 16 12:52:12.803958 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.800427    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"]
Sep 16 12:52:12.864271 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:12.864240    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s"]
Sep 16 12:52:12.867767 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:52:12.867744    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod773b717d_e15b_4aa9_a9fc_a61950b67915.slice/crio-395db38e0a35ddf4d4557b9ba690853c1705ab0eddb06e10f6615f52e5b9d67f WatchSource:0}: Error finding container 395db38e0a35ddf4d4557b9ba690853c1705ab0eddb06e10f6615f52e5b9d67f: Status 404 returned error can't find the container with id 395db38e0a35ddf4d4557b9ba690853c1705ab0eddb06e10f6615f52e5b9d67f
Sep 16 12:52:13.151967 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:13.151924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s" event={"ID":"773b717d-e15b-4aa9-a9fc-a61950b67915","Type":"ContainerStarted","Data":"395db38e0a35ddf4d4557b9ba690853c1705ab0eddb06e10f6615f52e5b9d67f"}
Sep 16 12:52:13.153019 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:13.152988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc" event={"ID":"5b27d374-cb23-4b62-867d-c4a34fbb235e","Type":"ContainerStarted","Data":"ccb92a7cc03ef71bbf7fb268e8b4425ded1508153e8927561ca4d435fd199f02"}
Sep 16 12:52:17.172873 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:17.172835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc" event={"ID":"5b27d374-cb23-4b62-867d-c4a34fbb235e","Type":"ContainerStarted","Data":"1b43bca5214a9e15e7383f044bbe278d2a8ddb165f6b72a0e0308662747e124f"}
Sep 16 12:52:17.173241 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:17.173019    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:17.174552 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:17.174530    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc"
Sep 16 12:52:17.191102 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:17.191059    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d46c9cfbc-66wbc" podStartSLOduration=1.567585061 podStartE2EDuration="5.191043729s" totalImagesPullingTime="3.623458668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:52:17.190217444 +0000 UTC m=+263.651846137" watchObservedRunningTime="2026-09-16 12:52:17.191043729 +0000 UTC m=+263.652672424"
Sep 16 12:52:27.209846 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:27.209768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s" event={"ID":"773b717d-e15b-4aa9-a9fc-a61950b67915","Type":"ContainerStarted","Data":"be62b975751df4295713d4661d2c5fda9a49f25be00948293e1f64fff84ad2c4"}
Sep 16 12:52:27.210256 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:27.209859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s" event={"ID":"773b717d-e15b-4aa9-a9fc-a61950b67915","Type":"ContainerStarted","Data":"eeeb9b0df4bf5765c6094e4615bf7a0a544f36afdca651b90db72b5925352c90"}
Sep 16 12:52:27.210256 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:27.209878    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s" event={"ID":"773b717d-e15b-4aa9-a9fc-a61950b67915","Type":"ContainerStarted","Data":"ce51542c8084ce8f799080a1a0eca8dd6118f2ccbda146a42ea0661eb900a8e9"}
Sep 16 12:52:27.230837 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:27.230777    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6c658485-wrr2s" podStartSLOduration=1.7334714679999998 podStartE2EDuration="15.230763628s" totalImagesPullingTime="13.49729216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:52:27.229020675 +0000 UTC m=+273.690649371" watchObservedRunningTime="2026-09-16 12:52:27.230763628 +0000 UTC m=+273.692392323"
Sep 16 12:52:53.961020 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:52:53.960990    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:54:47.110734 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.110703    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c8f769787-2dkcv"]
Sep 16 12:54:47.113669 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.113646    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.140913 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.140884    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8f769787-2dkcv"]
Sep 16 12:54:47.239629 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.239595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-service-ca\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.239801 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.239636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-oauth-serving-cert\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.239801 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.239653    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmpl\" (UniqueName: \"kubernetes.io/projected/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-kube-api-access-tqmpl\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.239801 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.239714    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-serving-cert\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.239801 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.239737    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-config\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.240019 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.239830    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-trusted-ca-bundle\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.240019 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.239883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-oauth-config\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341078 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341036    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-service-ca\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341216 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341093    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-oauth-serving-cert\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341216 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341110    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmpl\" (UniqueName: \"kubernetes.io/projected/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-kube-api-access-tqmpl\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341216 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-serving-cert\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341352 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-config\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341352 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-trusted-ca-bundle\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341452 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-oauth-config\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341988 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-oauth-serving-cert\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341988 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-config\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.341988 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.341984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-service-ca\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.342263 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.342233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-trusted-ca-bundle\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.343648 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.343626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-serving-cert\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.343749 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.343717    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-console-oauth-config\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.350571 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.350552    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmpl\" (UniqueName: \"kubernetes.io/projected/00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5-kube-api-access-tqmpl\") pod \"console-6c8f769787-2dkcv\" (UID: \"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5\") " pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.422367 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.422326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:47.697603 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:47.697534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8f769787-2dkcv"]
Sep 16 12:54:47.700892 ip-10-0-139-96 kubenswrapper[2558]: W0916 12:54:47.700867    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e9118b_e1b1_44c8_b4ab_94d95cc8f8e5.slice/crio-04b056cc8be177825cd80a4dfa8c4b6dba5039cee1ed524c0edf5b634a119fc1 WatchSource:0}: Error finding container 04b056cc8be177825cd80a4dfa8c4b6dba5039cee1ed524c0edf5b634a119fc1: Status 404 returned error can't find the container with id 04b056cc8be177825cd80a4dfa8c4b6dba5039cee1ed524c0edf5b634a119fc1
Sep 16 12:54:48.700341 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:48.700300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8f769787-2dkcv" event={"ID":"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5","Type":"ContainerStarted","Data":"b6f5f8883f2fb2dd0f52c4128afc2e2ec60371fbefb546b4f562e2b4057dab7c"}
Sep 16 12:54:48.700852 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:48.700358    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8f769787-2dkcv" event={"ID":"00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5","Type":"ContainerStarted","Data":"04b056cc8be177825cd80a4dfa8c4b6dba5039cee1ed524c0edf5b634a119fc1"}
Sep 16 12:54:48.720639 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:48.720593    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c8f769787-2dkcv" podStartSLOduration=1.720579071 podStartE2EDuration="1.720579071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:48.719424492 +0000 UTC m=+415.181053190" watchObservedRunningTime="2026-09-16 12:54:48.720579071 +0000 UTC m=+415.182207771"
Sep 16 12:54:57.422923 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:57.422873    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:57.423326 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:57.422955    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:57.427567 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:57.427544    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:57.735496 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:57.735417    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c8f769787-2dkcv"
Sep 16 12:54:57.795472 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:54:57.795441    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f7896f69-867wg"]
Sep 16 12:55:22.816252 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:22.816161    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77f7896f69-867wg" podUID="8ac9a191-da2a-4774-86fa-9f29a9659e38" containerName="console" containerID="cri-o://d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381" gracePeriod=15
Sep 16 12:55:23.051647 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.051626    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:55:23.077961 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.077887    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-oauth-serving-cert\") pod \"8ac9a191-da2a-4774-86fa-9f29a9659e38\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") "
Sep 16 12:55:23.077961 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.077932    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-config\") pod \"8ac9a191-da2a-4774-86fa-9f29a9659e38\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") "
Sep 16 12:55:23.078146 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.077991    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-oauth-config\") pod \"8ac9a191-da2a-4774-86fa-9f29a9659e38\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") "
Sep 16 12:55:23.078146 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078014    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-trusted-ca-bundle\") pod \"8ac9a191-da2a-4774-86fa-9f29a9659e38\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") "
Sep 16 12:55:23.078146 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078056    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-service-ca\") pod \"8ac9a191-da2a-4774-86fa-9f29a9659e38\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") "
Sep 16 12:55:23.078146 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078095    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbmzc\" (UniqueName: \"kubernetes.io/projected/8ac9a191-da2a-4774-86fa-9f29a9659e38-kube-api-access-gbmzc\") pod \"8ac9a191-da2a-4774-86fa-9f29a9659e38\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") "
Sep 16 12:55:23.078146 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078130    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-serving-cert\") pod \"8ac9a191-da2a-4774-86fa-9f29a9659e38\" (UID: \"8ac9a191-da2a-4774-86fa-9f29a9659e38\") "
Sep 16 12:55:23.078402 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8ac9a191-da2a-4774-86fa-9f29a9659e38" (UID: "8ac9a191-da2a-4774-86fa-9f29a9659e38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:23.078593 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-service-ca" (OuterVolumeSpecName: "service-ca") pod "8ac9a191-da2a-4774-86fa-9f29a9659e38" (UID: "8ac9a191-da2a-4774-86fa-9f29a9659e38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:23.078593 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078578    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8ac9a191-da2a-4774-86fa-9f29a9659e38" (UID: "8ac9a191-da2a-4774-86fa-9f29a9659e38"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:23.078746 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.078721    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-config" (OuterVolumeSpecName: "console-config") pod "8ac9a191-da2a-4774-86fa-9f29a9659e38" (UID: "8ac9a191-da2a-4774-86fa-9f29a9659e38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:23.080351 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.080329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8ac9a191-da2a-4774-86fa-9f29a9659e38" (UID: "8ac9a191-da2a-4774-86fa-9f29a9659e38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:55:23.080731 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.080709    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8ac9a191-da2a-4774-86fa-9f29a9659e38" (UID: "8ac9a191-da2a-4774-86fa-9f29a9659e38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:55:23.080731 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.080719    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac9a191-da2a-4774-86fa-9f29a9659e38-kube-api-access-gbmzc" (OuterVolumeSpecName: "kube-api-access-gbmzc") pod "8ac9a191-da2a-4774-86fa-9f29a9659e38" (UID: "8ac9a191-da2a-4774-86fa-9f29a9659e38"). InnerVolumeSpecName "kube-api-access-gbmzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:23.179436 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.179390    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmzc\" (UniqueName: \"kubernetes.io/projected/8ac9a191-da2a-4774-86fa-9f29a9659e38-kube-api-access-gbmzc\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:55:23.179436 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.179428    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:55:23.179436 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.179441    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-oauth-serving-cert\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:55:23.179679 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.179452    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:55:23.179679 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.179466    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ac9a191-da2a-4774-86fa-9f29a9659e38-console-oauth-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:55:23.179679 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.179481    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-trusted-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:55:23.179679 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.179491    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9a191-da2a-4774-86fa-9f29a9659e38-service-ca\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 12:55:23.827462 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.827424    2558 generic.go:356] "Generic (PLEG): container finished" podID="8ac9a191-da2a-4774-86fa-9f29a9659e38" containerID="d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381" exitCode=0
Sep 16 12:55:23.827913 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.827466    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f7896f69-867wg"
Sep 16 12:55:23.827913 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.827473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f7896f69-867wg" event={"ID":"8ac9a191-da2a-4774-86fa-9f29a9659e38","Type":"ContainerDied","Data":"d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381"}
Sep 16 12:55:23.827913 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.827522    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f7896f69-867wg" event={"ID":"8ac9a191-da2a-4774-86fa-9f29a9659e38","Type":"ContainerDied","Data":"3b3c5514b4cf3f2624d39d96cdde25d149558e52c9e27e38abce11f81d004a85"}
Sep 16 12:55:23.827913 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.827548    2558 scope.go:117] "RemoveContainer" containerID="d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381"
Sep 16 12:55:23.836738 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.836611    2558 scope.go:117] "RemoveContainer" containerID="d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381"
Sep 16 12:55:23.836958 ip-10-0-139-96 kubenswrapper[2558]: E0916 12:55:23.836933    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381\": container with ID starting with d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381 not found: ID does not exist" containerID="d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381"
Sep 16 12:55:23.837042 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.836970    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381"} err="failed to get container status \"d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381\": rpc error: code = NotFound desc = could not find container \"d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381\": container with ID starting with d50122a00da74a2ca8d127bb3c1109d52683d5789809ec502493a360e3adc381 not found: ID does not exist"
Sep 16 12:55:23.862951 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.862923    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f7896f69-867wg"]
Sep 16 12:55:23.873228 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:23.873202    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77f7896f69-867wg"]
Sep 16 12:55:24.041210 ip-10-0-139-96 kubenswrapper[2558]: I0916 12:55:24.041177    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac9a191-da2a-4774-86fa-9f29a9659e38" path="/var/lib/kubelet/pods/8ac9a191-da2a-4774-86fa-9f29a9659e38/volumes"
Sep 16 13:00:33.866659 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.866620    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-7jj7p"]
Sep 16 13:00:33.867175 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.866886    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac9a191-da2a-4774-86fa-9f29a9659e38" containerName="console"
Sep 16 13:00:33.867175 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.866897    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac9a191-da2a-4774-86fa-9f29a9659e38" containerName="console"
Sep 16 13:00:33.867175 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.866942    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac9a191-da2a-4774-86fa-9f29a9659e38" containerName="console"
Sep 16 13:00:33.869617 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.869602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:33.872409 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.872383    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 13:00:33.872566 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.872480    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-spnxc\""
Sep 16 13:00:33.873641 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.873623    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 13:00:33.873752 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.873682    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 13:00:33.891401 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.891370    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7jj7p"]
Sep 16 13:00:33.893067 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.893043    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-nk86t"]
Sep 16 13:00:33.896114 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.896094    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:33.905341 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.905320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nk86t"]
Sep 16 13:00:33.914060 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.914040    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 13:00:33.914182 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.914040    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-2fzrt\""
Sep 16 13:00:33.914182 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:33.914113    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 13:00:34.014687 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.014652    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f813514f-6a80-45b0-84d3-c8d72fa03762-metrics-certs\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.014894 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.014702    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044a3d0f-607e-4ff9-8c65-83fc6b71fd17-tls-certs\") pod \"model-serving-api-7b954ffc95-7jj7p\" (UID: \"044a3d0f-607e-4ff9-8c65-83fc6b71fd17\") " pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:34.014894 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.014721    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvq8h\" (UniqueName: \"kubernetes.io/projected/044a3d0f-607e-4ff9-8c65-83fc6b71fd17-kube-api-access-cvq8h\") pod \"model-serving-api-7b954ffc95-7jj7p\" (UID: \"044a3d0f-607e-4ff9-8c65-83fc6b71fd17\") " pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:34.014894 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.014786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f813514f-6a80-45b0-84d3-c8d72fa03762-cert\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.014894 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.014872    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4w8\" (UniqueName: \"kubernetes.io/projected/f813514f-6a80-45b0-84d3-c8d72fa03762-kube-api-access-2w4w8\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.115479 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.115441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f813514f-6a80-45b0-84d3-c8d72fa03762-metrics-certs\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.115674 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.115495    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044a3d0f-607e-4ff9-8c65-83fc6b71fd17-tls-certs\") pod \"model-serving-api-7b954ffc95-7jj7p\" (UID: \"044a3d0f-607e-4ff9-8c65-83fc6b71fd17\") " pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:34.115674 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.115511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvq8h\" (UniqueName: \"kubernetes.io/projected/044a3d0f-607e-4ff9-8c65-83fc6b71fd17-kube-api-access-cvq8h\") pod \"model-serving-api-7b954ffc95-7jj7p\" (UID: \"044a3d0f-607e-4ff9-8c65-83fc6b71fd17\") " pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:34.115674 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.115527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f813514f-6a80-45b0-84d3-c8d72fa03762-cert\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.115674 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.115582    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4w8\" (UniqueName: \"kubernetes.io/projected/f813514f-6a80-45b0-84d3-c8d72fa03762-kube-api-access-2w4w8\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.118088 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.118034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f813514f-6a80-45b0-84d3-c8d72fa03762-metrics-certs\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.118178 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.118096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f813514f-6a80-45b0-84d3-c8d72fa03762-cert\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.118178 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.118143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044a3d0f-607e-4ff9-8c65-83fc6b71fd17-tls-certs\") pod \"model-serving-api-7b954ffc95-7jj7p\" (UID: \"044a3d0f-607e-4ff9-8c65-83fc6b71fd17\") " pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:34.125889 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.125863    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvq8h\" (UniqueName: \"kubernetes.io/projected/044a3d0f-607e-4ff9-8c65-83fc6b71fd17-kube-api-access-cvq8h\") pod \"model-serving-api-7b954ffc95-7jj7p\" (UID: \"044a3d0f-607e-4ff9-8c65-83fc6b71fd17\") " pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:34.131346 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.131320    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4w8\" (UniqueName: \"kubernetes.io/projected/f813514f-6a80-45b0-84d3-c8d72fa03762-kube-api-access-2w4w8\") pod \"odh-model-controller-7b966d7984-nk86t\" (UID: \"f813514f-6a80-45b0-84d3-c8d72fa03762\") " pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.178908 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.178877    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:34.206874 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.206845    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:34.465212 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.465175    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7jj7p"]
Sep 16 13:00:34.470171 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:00:34.470124    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044a3d0f_607e_4ff9_8c65_83fc6b71fd17.slice/crio-09b213e76e817da5a07dca26a354cf86e56a78d4307631c37b70237c9f0dd10a WatchSource:0}: Error finding container 09b213e76e817da5a07dca26a354cf86e56a78d4307631c37b70237c9f0dd10a: Status 404 returned error can't find the container with id 09b213e76e817da5a07dca26a354cf86e56a78d4307631c37b70237c9f0dd10a
Sep 16 13:00:34.472040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.472022    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:00:34.504053 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.504027    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nk86t"]
Sep 16 13:00:34.506947 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:00:34.506916    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf813514f_6a80_45b0_84d3_c8d72fa03762.slice/crio-b9840eacd6dfcc8c6ec3695d8701d979990b917b310493db807d6bf8e4ba0405 WatchSource:0}: Error finding container b9840eacd6dfcc8c6ec3695d8701d979990b917b310493db807d6bf8e4ba0405: Status 404 returned error can't find the container with id b9840eacd6dfcc8c6ec3695d8701d979990b917b310493db807d6bf8e4ba0405
Sep 16 13:00:34.902287 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.902253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nk86t" event={"ID":"f813514f-6a80-45b0-84d3-c8d72fa03762","Type":"ContainerStarted","Data":"b9840eacd6dfcc8c6ec3695d8701d979990b917b310493db807d6bf8e4ba0405"}
Sep 16 13:00:34.903552 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:34.903527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7jj7p" event={"ID":"044a3d0f-607e-4ff9-8c65-83fc6b71fd17","Type":"ContainerStarted","Data":"09b213e76e817da5a07dca26a354cf86e56a78d4307631c37b70237c9f0dd10a"}
Sep 16 13:00:38.922575 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:38.922535    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7jj7p" event={"ID":"044a3d0f-607e-4ff9-8c65-83fc6b71fd17","Type":"ContainerStarted","Data":"85b0d54bebde19b1fceb066105469cf16e3e9ac98c4ac60f28ad43df99f3c28c"}
Sep 16 13:00:38.923041 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:38.922673    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:00:38.923974 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:38.923951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nk86t" event={"ID":"f813514f-6a80-45b0-84d3-c8d72fa03762","Type":"ContainerStarted","Data":"65d2bb196133f635b0e56bea27b5463502e4b714d34e8d1f6041177d20eeb5b3"}
Sep 16 13:00:38.924143 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:38.924130    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:38.947255 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:38.947178    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-7jj7p" podStartSLOduration=2.229385236 podStartE2EDuration="5.947162174s" totalImagesPullingTime="3.717776938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:38.94499086 +0000 UTC m=+765.406619555" watchObservedRunningTime="2026-09-16 13:00:38.947162174 +0000 UTC m=+765.408790870"
Sep 16 13:00:38.966939 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:38.966780    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-nk86t" podStartSLOduration=2.322900825 podStartE2EDuration="5.966761567s" totalImagesPullingTime="3.643860742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:38.965898112 +0000 UTC m=+765.427526809" watchObservedRunningTime="2026-09-16 13:00:38.966761567 +0000 UTC m=+765.428390249"
Sep 16 13:00:49.930194 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:49.930163    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-nk86t"
Sep 16 13:00:49.932120 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:00:49.932095    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-7jj7p"
Sep 16 13:02:21.573798 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.573758    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"]
Sep 16 13:02:21.576874 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.576856    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.579745 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.579721    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 13:02:21.579890 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.579760    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-18839-predictor-serving-cert\""
Sep 16 13:02:21.579890 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.579839    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-18839-kube-rbac-proxy-sar-config\""
Sep 16 13:02:21.580139 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.580123    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 13:02:21.580196 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.580161    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gdsnl\""
Sep 16 13:02:21.587867 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.587846    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"]
Sep 16 13:02:21.630689 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.630653    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6957p\" (UniqueName: \"kubernetes.io/projected/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-api-access-6957p\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.630846 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.630698    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.630846 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.630761    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e1a540-7df7-40ab-bd62-47ca77a1b932-proxy-tls\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.731332 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.731295    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6957p\" (UniqueName: \"kubernetes.io/projected/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-api-access-6957p\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.731483 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.731352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.731483 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.731397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e1a540-7df7-40ab-bd62-47ca77a1b932-proxy-tls\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.732149 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.732129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.733746 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.733727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e1a540-7df7-40ab-bd62-47ca77a1b932-proxy-tls\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.739006 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.738983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6957p\" (UniqueName: \"kubernetes.io/projected/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-api-access-6957p\") pod \"error-404-isvc-18839-predictor-7c655d78b4-v8rzm\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") " pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:21.888547 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:21.888461    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:22.171131 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:22.171098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"]
Sep 16 13:02:22.279972 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:22.279931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" event={"ID":"69e1a540-7df7-40ab-bd62-47ca77a1b932","Type":"ContainerStarted","Data":"07fe357b2c8fe921e29bcbdc4748c1864ee5136c68b117ae4b356652e2266ea1"}
Sep 16 13:02:34.333689 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:34.333605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" event={"ID":"69e1a540-7df7-40ab-bd62-47ca77a1b932","Type":"ContainerStarted","Data":"9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc"}
Sep 16 13:02:37.347044 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:37.347003    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" event={"ID":"69e1a540-7df7-40ab-bd62-47ca77a1b932","Type":"ContainerStarted","Data":"1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee"}
Sep 16 13:02:37.347500 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:37.347091    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:37.347500 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:37.347141    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:37.348460 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:37.348437    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 13:02:37.364730 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:37.364679    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podStartSLOduration=2.21680883 podStartE2EDuration="16.364666075s" totalImagesPullingTime="14.147857245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:37.363494223 +0000 UTC m=+883.825122920" watchObservedRunningTime="2026-09-16 13:02:37.364666075 +0000 UTC m=+883.826294772"
Sep 16 13:02:38.351403 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:38.351359    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 13:02:43.356160 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:43.356130    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:02:43.356788 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:43.356755    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 13:02:53.357094 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:02:53.357010    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 13:03:03.357496 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:03.357452    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 13:03:13.357315 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:13.357270    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 13:03:23.357913 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:23.357881    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:03:41.169751 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.169717    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"]
Sep 16 13:03:41.172898 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.172880    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.175333 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.175312    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-18839-kube-rbac-proxy-sar-config\""
Sep 16 13:03:41.175447 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.175345    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-18839-serving-cert\""
Sep 16 13:03:41.183971 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.183949    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"]
Sep 16 13:03:41.264700 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.264666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-proxy-tls\") pod \"switch-graph-18839-5b789bf7-khdjv\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") " pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.264893 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.264734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-openshift-service-ca-bundle\") pod \"switch-graph-18839-5b789bf7-khdjv\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") " pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.365326 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.365292    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-proxy-tls\") pod \"switch-graph-18839-5b789bf7-khdjv\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") " pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.365488 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.365350    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-openshift-service-ca-bundle\") pod \"switch-graph-18839-5b789bf7-khdjv\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") " pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.366000 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.365982    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-openshift-service-ca-bundle\") pod \"switch-graph-18839-5b789bf7-khdjv\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") " pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.367603 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.367584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-proxy-tls\") pod \"switch-graph-18839-5b789bf7-khdjv\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") " pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.483000 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.482916    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:41.753371 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:41.753269    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"]
Sep 16 13:03:41.755757 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:03:41.755731    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a84274_b55d_4dbd_9c3a_9fe7e5e5aaa1.slice/crio-792aecaae2385acc854d0e03268a893246f1794a073086f266f0c6beaaf52b6c WatchSource:0}: Error finding container 792aecaae2385acc854d0e03268a893246f1794a073086f266f0c6beaaf52b6c: Status 404 returned error can't find the container with id 792aecaae2385acc854d0e03268a893246f1794a073086f266f0c6beaaf52b6c
Sep 16 13:03:42.580640 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:42.580586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" event={"ID":"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1","Type":"ContainerStarted","Data":"792aecaae2385acc854d0e03268a893246f1794a073086f266f0c6beaaf52b6c"}
Sep 16 13:03:44.589574 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:44.589487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" event={"ID":"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1","Type":"ContainerStarted","Data":"d6b50c7c39b917d4494d8809aad45dfb50a66599ddf20cac772f78e22c2ae7c3"}
Sep 16 13:03:44.589574 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:44.589549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:44.606411 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:44.606367    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podStartSLOduration=1.053313042 podStartE2EDuration="3.606351707s" totalImagesPullingTime="2.553038665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:03:44.605946176 +0000 UTC m=+951.067574873" watchObservedRunningTime="2026-09-16 13:03:44.606351707 +0000 UTC m=+951.067980403"
Sep 16 13:03:50.601499 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:50.601469    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:03:51.283303 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.283270    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"]
Sep 16 13:03:51.283493 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.283455    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839" containerID="cri-o://d6b50c7c39b917d4494d8809aad45dfb50a66599ddf20cac772f78e22c2ae7c3" gracePeriod=30
Sep 16 13:03:51.396063 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.396028    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"]
Sep 16 13:03:51.396306 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.396271    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" containerID="cri-o://9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc" gracePeriod=30
Sep 16 13:03:51.396472 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.396292    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kube-rbac-proxy" containerID="cri-o://1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee" gracePeriod=30
Sep 16 13:03:51.572749 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.572677    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"]
Sep 16 13:03:51.575717 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.575701    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.578195 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.578174    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-03a3a-predictor-serving-cert\""
Sep 16 13:03:51.578305 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.578196    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-03a3a-kube-rbac-proxy-sar-config\""
Sep 16 13:03:51.584880 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.584858    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"]
Sep 16 13:03:51.619234 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.619201    2558 generic.go:356] "Generic (PLEG): container finished" podID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerID="1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee" exitCode=2
Sep 16 13:03:51.619584 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.619246    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" event={"ID":"69e1a540-7df7-40ab-bd62-47ca77a1b932","Type":"ContainerDied","Data":"1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee"}
Sep 16 13:03:51.647636 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.647601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82chl\" (UniqueName: \"kubernetes.io/projected/9065332b-0aad-41ac-ac58-5924cfea2119-kube-api-access-82chl\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.647772 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.647647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9065332b-0aad-41ac-ac58-5924cfea2119-proxy-tls\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.647772 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.647715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9065332b-0aad-41ac-ac58-5924cfea2119-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.748419 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.748382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9065332b-0aad-41ac-ac58-5924cfea2119-proxy-tls\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.748584 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.748438    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9065332b-0aad-41ac-ac58-5924cfea2119-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.748584 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.748510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82chl\" (UniqueName: \"kubernetes.io/projected/9065332b-0aad-41ac-ac58-5924cfea2119-kube-api-access-82chl\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.749147 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.749122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9065332b-0aad-41ac-ac58-5924cfea2119-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.751011 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.750983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9065332b-0aad-41ac-ac58-5924cfea2119-proxy-tls\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.756608 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.756585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82chl\" (UniqueName: \"kubernetes.io/projected/9065332b-0aad-41ac-ac58-5924cfea2119-kube-api-access-82chl\") pod \"error-404-isvc-03a3a-predictor-864f8dfd-tx9rm\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") " pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:51.886222 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:51.886140    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:52.156517 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:52.156486    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"]
Sep 16 13:03:52.160615 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:03:52.160587    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9065332b_0aad_41ac_ac58_5924cfea2119.slice/crio-ec30979f16a5dafad84f689da483790983a4aeaf2be383e9c546c47f54903a16 WatchSource:0}: Error finding container ec30979f16a5dafad84f689da483790983a4aeaf2be383e9c546c47f54903a16: Status 404 returned error can't find the container with id ec30979f16a5dafad84f689da483790983a4aeaf2be383e9c546c47f54903a16
Sep 16 13:03:52.624995 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:52.624899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" event={"ID":"9065332b-0aad-41ac-ac58-5924cfea2119","Type":"ContainerStarted","Data":"0674ff8f45ce08926aefbe54ff7daf40040319d31655bf3cecfda722f94e7179"}
Sep 16 13:03:52.624995 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:52.624968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" event={"ID":"9065332b-0aad-41ac-ac58-5924cfea2119","Type":"ContainerStarted","Data":"81f8d335999e7ba557b606e5ee698d2b0e61343f9f85fd545460a847de005204"}
Sep 16 13:03:52.625497 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:52.625000    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:52.625497 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:52.625029    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" event={"ID":"9065332b-0aad-41ac-ac58-5924cfea2119","Type":"ContainerStarted","Data":"ec30979f16a5dafad84f689da483790983a4aeaf2be383e9c546c47f54903a16"}
Sep 16 13:03:52.642681 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:52.642631    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podStartSLOduration=1.642614086 podStartE2EDuration="1.642614086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:03:52.641985641 +0000 UTC m=+959.103614338" watchObservedRunningTime="2026-09-16 13:03:52.642614086 +0000 UTC m=+959.104242782"
Sep 16 13:03:53.352634 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:53.352588    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 13:03:53.357095 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:53.357063    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 13:03:53.628898 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:53.628776    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:53.630068 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:53.630041    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 13:03:54.443554 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.443528    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:03:54.574319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.574236    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-rbac-proxy-sar-config\") pod \"69e1a540-7df7-40ab-bd62-47ca77a1b932\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") "
Sep 16 13:03:54.574319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.574294    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6957p\" (UniqueName: \"kubernetes.io/projected/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-api-access-6957p\") pod \"69e1a540-7df7-40ab-bd62-47ca77a1b932\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") "
Sep 16 13:03:54.574319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.574312    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e1a540-7df7-40ab-bd62-47ca77a1b932-proxy-tls\") pod \"69e1a540-7df7-40ab-bd62-47ca77a1b932\" (UID: \"69e1a540-7df7-40ab-bd62-47ca77a1b932\") "
Sep 16 13:03:54.574631 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.574606    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69e1a540-7df7-40ab-bd62-47ca77a1b932" (UID: "69e1a540-7df7-40ab-bd62-47ca77a1b932"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:54.576455 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.576425    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-api-access-6957p" (OuterVolumeSpecName: "kube-api-access-6957p") pod "69e1a540-7df7-40ab-bd62-47ca77a1b932" (UID: "69e1a540-7df7-40ab-bd62-47ca77a1b932"). InnerVolumeSpecName "kube-api-access-6957p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:54.576556 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.576510    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e1a540-7df7-40ab-bd62-47ca77a1b932-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69e1a540-7df7-40ab-bd62-47ca77a1b932" (UID: "69e1a540-7df7-40ab-bd62-47ca77a1b932"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:03:54.633866 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.633838    2558 generic.go:356] "Generic (PLEG): container finished" podID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerID="9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc" exitCode=0
Sep 16 13:03:54.634225 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.633918    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"
Sep 16 13:03:54.634225 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.633964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" event={"ID":"69e1a540-7df7-40ab-bd62-47ca77a1b932","Type":"ContainerDied","Data":"9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc"}
Sep 16 13:03:54.634225 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.634028    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm" event={"ID":"69e1a540-7df7-40ab-bd62-47ca77a1b932","Type":"ContainerDied","Data":"07fe357b2c8fe921e29bcbdc4748c1864ee5136c68b117ae4b356652e2266ea1"}
Sep 16 13:03:54.634225 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.634058    2558 scope.go:117] "RemoveContainer" containerID="1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee"
Sep 16 13:03:54.634652 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.634620    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 13:03:54.642380 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.642359    2558 scope.go:117] "RemoveContainer" containerID="9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc"
Sep 16 13:03:54.649337 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.649318    2558 scope.go:117] "RemoveContainer" containerID="1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee"
Sep 16 13:03:54.649564 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:03:54.649547    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee\": container with ID starting with 1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee not found: ID does not exist" containerID="1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee"
Sep 16 13:03:54.649605 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.649572    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee"} err="failed to get container status \"1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee\": rpc error: code = NotFound desc = could not find container \"1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee\": container with ID starting with 1a6dd41b7a0bbcaadc2fd0485099ce1d56bf849c15c839b70dcb1714760756ee not found: ID does not exist"
Sep 16 13:03:54.649605 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.649592    2558 scope.go:117] "RemoveContainer" containerID="9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc"
Sep 16 13:03:54.649794 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:03:54.649779    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc\": container with ID starting with 9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc not found: ID does not exist" containerID="9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc"
Sep 16 13:03:54.649860 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.649798    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc"} err="failed to get container status \"9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc\": rpc error: code = NotFound desc = could not find container \"9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc\": container with ID starting with 9ea00cdee4a9a916f297c2d6dc41119c8a3d00727641e938e2a83be2fd5219dc not found: ID does not exist"
Sep 16 13:03:54.658344 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.658320    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"]
Sep 16 13:03:54.661100 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.661080    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18839-predictor-7c655d78b4-v8rzm"]
Sep 16 13:03:54.675057 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.675031    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e1a540-7df7-40ab-bd62-47ca77a1b932-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:03:54.675057 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.675055    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:03:54.675181 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:54.675070    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6957p\" (UniqueName: \"kubernetes.io/projected/69e1a540-7df7-40ab-bd62-47ca77a1b932-kube-api-access-6957p\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:03:55.600803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:55.600695    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:03:56.042145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:56.042113    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" path="/var/lib/kubelet/pods/69e1a540-7df7-40ab-bd62-47ca77a1b932/volumes"
Sep 16 13:03:59.640223 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:59.640194    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:03:59.640829 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:03:59.640789    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 13:04:00.600675 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:00.600633    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:05.600369 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:05.600332    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:05.600775 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:05.600459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:04:09.640869 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:09.640804    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 13:04:10.600148 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:10.600110    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:15.599761 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:15.599707    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:19.641096 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:19.641018    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 13:04:20.600437 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:20.600401    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:21.735748 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:21.735717    2558 generic.go:356] "Generic (PLEG): container finished" podID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerID="d6b50c7c39b917d4494d8809aad45dfb50a66599ddf20cac772f78e22c2ae7c3" exitCode=0
Sep 16 13:04:21.736132 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:21.735763    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" event={"ID":"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1","Type":"ContainerDied","Data":"d6b50c7c39b917d4494d8809aad45dfb50a66599ddf20cac772f78e22c2ae7c3"}
Sep 16 13:04:21.937495 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:21.937472    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:04:22.083781 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.083700    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-proxy-tls\") pod \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") "
Sep 16 13:04:22.083935 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.083783    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-openshift-service-ca-bundle\") pod \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\" (UID: \"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1\") "
Sep 16 13:04:22.084117 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.084089    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" (UID: "c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:04:22.085762 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.085735    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" (UID: "c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:04:22.184972 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.184938    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:04:22.184972 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.184966    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:04:22.740328 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.740300    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"
Sep 16 13:04:22.740760 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.740299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv" event={"ID":"c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1","Type":"ContainerDied","Data":"792aecaae2385acc854d0e03268a893246f1794a073086f266f0c6beaaf52b6c"}
Sep 16 13:04:22.740760 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.740439    2558 scope.go:117] "RemoveContainer" containerID="d6b50c7c39b917d4494d8809aad45dfb50a66599ddf20cac772f78e22c2ae7c3"
Sep 16 13:04:22.761104 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.761070    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"]
Sep 16 13:04:22.762774 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:22.762754    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-18839-5b789bf7-khdjv"]
Sep 16 13:04:24.041079 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:24.041048    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" path="/var/lib/kubelet/pods/c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1/volumes"
Sep 16 13:04:29.641099 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:29.641059    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 13:04:39.641528 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:39.641496    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:04:41.142635 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142597    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"]
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142886    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142899    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142917    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kube-rbac-proxy"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142922    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kube-rbac-proxy"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142933    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142940    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142983    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kube-rbac-proxy"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.142991    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a84274-b55d-4dbd-9c3a-9fe7e5e5aaa1" containerName="switch-graph-18839"
Sep 16 13:04:41.143040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.143000    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e1a540-7df7-40ab-bd62-47ca77a1b932" containerName="kserve-container"
Sep 16 13:04:41.147154 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.147135    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:41.149640 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.149617    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 16 13:04:41.149750 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.149651    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 16 13:04:41.155007 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.154986    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"]
Sep 16 13:04:41.220339 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.220307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls\") pod \"model-chainer-6dd9749d5-24jpf\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") " pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:41.220502 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.220363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-openshift-service-ca-bundle\") pod \"model-chainer-6dd9749d5-24jpf\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") " pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:41.321519 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.321479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls\") pod \"model-chainer-6dd9749d5-24jpf\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") " pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:41.321701 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.321563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-openshift-service-ca-bundle\") pod \"model-chainer-6dd9749d5-24jpf\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") " pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:41.321701 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:04:41.321630    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Sep 16 13:04:41.321845 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:04:41.321709    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls podName:6202933f-18cb-4c1a-b6a8-bd11c4159ba9 nodeName:}" failed. No retries permitted until 2026-09-16 13:04:41.821689761 +0000 UTC m=+1008.283318435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls") pod "model-chainer-6dd9749d5-24jpf" (UID: "6202933f-18cb-4c1a-b6a8-bd11c4159ba9") : secret "model-chainer-serving-cert" not found
Sep 16 13:04:41.322195 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.322174    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-openshift-service-ca-bundle\") pod \"model-chainer-6dd9749d5-24jpf\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") " pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:41.824561 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.824528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls\") pod \"model-chainer-6dd9749d5-24jpf\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") " pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:41.826804 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:41.826787    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls\") pod \"model-chainer-6dd9749d5-24jpf\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") " pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:42.057139 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:42.057109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:42.325978 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:42.325947    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"]
Sep 16 13:04:42.330087 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:04:42.330059    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6202933f_18cb_4c1a_b6a8_bd11c4159ba9.slice/crio-42905e2ee7893353de6f1beeb34440765ec294cab9e0b3792bd376fd6f40b5ff WatchSource:0}: Error finding container 42905e2ee7893353de6f1beeb34440765ec294cab9e0b3792bd376fd6f40b5ff: Status 404 returned error can't find the container with id 42905e2ee7893353de6f1beeb34440765ec294cab9e0b3792bd376fd6f40b5ff
Sep 16 13:04:42.821726 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:42.821692    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" event={"ID":"6202933f-18cb-4c1a-b6a8-bd11c4159ba9","Type":"ContainerStarted","Data":"c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90"}
Sep 16 13:04:42.821934 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:42.821745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" event={"ID":"6202933f-18cb-4c1a-b6a8-bd11c4159ba9","Type":"ContainerStarted","Data":"42905e2ee7893353de6f1beeb34440765ec294cab9e0b3792bd376fd6f40b5ff"}
Sep 16 13:04:42.821934 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:42.821844    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:42.839833 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:42.839764    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podStartSLOduration=1.8397488100000001 podStartE2EDuration="1.83974881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:04:42.838032985 +0000 UTC m=+1009.299661684" watchObservedRunningTime="2026-09-16 13:04:42.83974881 +0000 UTC m=+1009.301377506"
Sep 16 13:04:48.837413 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:48.837382    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:04:51.255364 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.255325    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"]
Sep 16 13:04:51.255845 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.255629    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer" containerID="cri-o://c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90" gracePeriod=30
Sep 16 13:04:51.493065 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.493029    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"]
Sep 16 13:04:51.496368 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.496351    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.498988 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.498952    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-03a3a-kube-rbac-proxy-sar-config\""
Sep 16 13:04:51.498988 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.498984    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-03a3a-serving-cert\""
Sep 16 13:04:51.507238 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.507186    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"]
Sep 16 13:04:51.590537 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.590507    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"]
Sep 16 13:04:51.593667 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.593649    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.596501 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.596482    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-46b60-predictor-serving-cert\""
Sep 16 13:04:51.596630 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.596504    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-46b60-kube-rbac-proxy-sar-config\""
Sep 16 13:04:51.597431 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.597408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7013f247-3f5c-40d5-a59b-e2ed07222a95-openshift-service-ca-bundle\") pod \"switch-graph-03a3a-5ff944788f-79jqg\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") " pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.597551 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.597444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7013f247-3f5c-40d5-a59b-e2ed07222a95-proxy-tls\") pod \"switch-graph-03a3a-5ff944788f-79jqg\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") " pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.604119 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.604096    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"]
Sep 16 13:04:51.697824 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.697794    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b0-6e22-42b0-aa77-99e87d3a4354-proxy-tls\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.697981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.697858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.697981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.697947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7013f247-3f5c-40d5-a59b-e2ed07222a95-openshift-service-ca-bundle\") pod \"switch-graph-03a3a-5ff944788f-79jqg\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") " pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.698059 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.697980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7013f247-3f5c-40d5-a59b-e2ed07222a95-proxy-tls\") pod \"switch-graph-03a3a-5ff944788f-79jqg\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") " pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.698059 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.697999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccsc\" (UniqueName: \"kubernetes.io/projected/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-api-access-hccsc\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.698533 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.698517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7013f247-3f5c-40d5-a59b-e2ed07222a95-openshift-service-ca-bundle\") pod \"switch-graph-03a3a-5ff944788f-79jqg\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") " pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.700317 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.700296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7013f247-3f5c-40d5-a59b-e2ed07222a95-proxy-tls\") pod \"switch-graph-03a3a-5ff944788f-79jqg\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") " pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.799166 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.799074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.799166 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.799130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccsc\" (UniqueName: \"kubernetes.io/projected/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-api-access-hccsc\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.799166 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.799169    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b0-6e22-42b0-aa77-99e87d3a4354-proxy-tls\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.799782 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.799754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.801530 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.801507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b0-6e22-42b0-aa77-99e87d3a4354-proxy-tls\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.806137 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.806111    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:51.807016 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.806992    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccsc\" (UniqueName: \"kubernetes.io/projected/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-api-access-hccsc\") pod \"error-404-isvc-46b60-predictor-786fd95898-n46zk\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") " pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:51.904084 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:51.904049    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:52.073925 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.073901    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"]
Sep 16 13:04:52.076425 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:04:52.076395    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7013f247_3f5c_40d5_a59b_e2ed07222a95.slice/crio-b4855b9cfaeca65174418ac76282cd497c004143d47be491efa7c9c60530eb17 WatchSource:0}: Error finding container b4855b9cfaeca65174418ac76282cd497c004143d47be491efa7c9c60530eb17: Status 404 returned error can't find the container with id b4855b9cfaeca65174418ac76282cd497c004143d47be491efa7c9c60530eb17
Sep 16 13:04:52.170394 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.170370    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"]
Sep 16 13:04:52.172718 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:04:52.172693    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4861b0_6e22_42b0_aa77_99e87d3a4354.slice/crio-2ebab2874c5182da55d5484aa32b841806bcff63459944d7b518c214127c074e WatchSource:0}: Error finding container 2ebab2874c5182da55d5484aa32b841806bcff63459944d7b518c214127c074e: Status 404 returned error can't find the container with id 2ebab2874c5182da55d5484aa32b841806bcff63459944d7b518c214127c074e
Sep 16 13:04:52.867934 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.867898    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" event={"ID":"7013f247-3f5c-40d5-a59b-e2ed07222a95","Type":"ContainerStarted","Data":"93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314"}
Sep 16 13:04:52.868392 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.867959    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" event={"ID":"7013f247-3f5c-40d5-a59b-e2ed07222a95","Type":"ContainerStarted","Data":"b4855b9cfaeca65174418ac76282cd497c004143d47be491efa7c9c60530eb17"}
Sep 16 13:04:52.868392 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.868074    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:52.869676 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.869652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" event={"ID":"fa4861b0-6e22-42b0-aa77-99e87d3a4354","Type":"ContainerStarted","Data":"7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc"}
Sep 16 13:04:52.869797 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.869696    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" event={"ID":"fa4861b0-6e22-42b0-aa77-99e87d3a4354","Type":"ContainerStarted","Data":"bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35"}
Sep 16 13:04:52.869797 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.869722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" event={"ID":"fa4861b0-6e22-42b0-aa77-99e87d3a4354","Type":"ContainerStarted","Data":"2ebab2874c5182da55d5484aa32b841806bcff63459944d7b518c214127c074e"}
Sep 16 13:04:52.869797 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.869745    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:52.884000 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.883952    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podStartSLOduration=1.883938557 podStartE2EDuration="1.883938557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:04:52.883415507 +0000 UTC m=+1019.345044206" watchObservedRunningTime="2026-09-16 13:04:52.883938557 +0000 UTC m=+1019.345567251"
Sep 16 13:04:52.900170 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:52.900127    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podStartSLOduration=1.900113174 podStartE2EDuration="1.900113174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:04:52.89930996 +0000 UTC m=+1019.360938657" watchObservedRunningTime="2026-09-16 13:04:52.900113174 +0000 UTC m=+1019.361741870"
Sep 16 13:04:53.835743 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:53.835707    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:53.874623 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:53.874583    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:53.875755 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:53.875726    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 13:04:54.878386 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:54.878344    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 13:04:58.835729 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:58.835687    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:58.879709 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:58.879680    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:04:59.883390 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:59.883356    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:04:59.883922 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:04:59.883895    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 13:05:03.835979 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:03.835933    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:05:03.836447 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:03.836039    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:05:08.835656 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:08.835615    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:05:09.884003 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:09.883963    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 13:05:13.835434 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:13.835399    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:05:18.835517 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:18.835476    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:05:19.884401 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:19.884364    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 13:05:21.393148 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.393125    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:05:21.564508 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.564418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-openshift-service-ca-bundle\") pod \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") "
Sep 16 13:05:21.564508 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.564506    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls\") pod \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\" (UID: \"6202933f-18cb-4c1a-b6a8-bd11c4159ba9\") "
Sep 16 13:05:21.564804 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.564774    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6202933f-18cb-4c1a-b6a8-bd11c4159ba9" (UID: "6202933f-18cb-4c1a-b6a8-bd11c4159ba9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:21.566610 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.566583    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6202933f-18cb-4c1a-b6a8-bd11c4159ba9" (UID: "6202933f-18cb-4c1a-b6a8-bd11c4159ba9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:21.666024 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.665986    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:05:21.666024 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.666017    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6202933f-18cb-4c1a-b6a8-bd11c4159ba9-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:05:21.976754 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.976717    2558 generic.go:356] "Generic (PLEG): container finished" podID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerID="c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90" exitCode=0
Sep 16 13:05:21.976949 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.976774    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"
Sep 16 13:05:21.976949 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.976781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" event={"ID":"6202933f-18cb-4c1a-b6a8-bd11c4159ba9","Type":"ContainerDied","Data":"c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90"}
Sep 16 13:05:21.976949 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.976869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf" event={"ID":"6202933f-18cb-4c1a-b6a8-bd11c4159ba9","Type":"ContainerDied","Data":"42905e2ee7893353de6f1beeb34440765ec294cab9e0b3792bd376fd6f40b5ff"}
Sep 16 13:05:21.976949 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.976892    2558 scope.go:117] "RemoveContainer" containerID="c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90"
Sep 16 13:05:21.985904 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.985708    2558 scope.go:117] "RemoveContainer" containerID="c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90"
Sep 16 13:05:21.986171 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:05:21.986150    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90\": container with ID starting with c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90 not found: ID does not exist" containerID="c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90"
Sep 16 13:05:21.986244 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.986184    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90"} err="failed to get container status \"c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90\": rpc error: code = NotFound desc = could not find container \"c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90\": container with ID starting with c48cb5d9acda88ed60715a0b0ef6dfa94e9a3046414e6147dccfe5888df32e90 not found: ID does not exist"
Sep 16 13:05:21.998928 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:21.998901    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"]
Sep 16 13:05:22.001720 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:22.001699    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6dd9749d5-24jpf"]
Sep 16 13:05:22.041732 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:22.041693    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" path="/var/lib/kubelet/pods/6202933f-18cb-4c1a-b6a8-bd11c4159ba9/volumes"
Sep 16 13:05:29.884208 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:29.884167    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 13:05:39.884791 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:39.884762    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:05:51.465655 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.465573    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"]
Sep 16 13:05:51.466065 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.466046    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer"
Sep 16 13:05:51.466112 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.466070    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer"
Sep 16 13:05:51.466181 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.466170    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6202933f-18cb-4c1a-b6a8-bd11c4159ba9" containerName="model-chainer"
Sep 16 13:05:51.470243 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.470225    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:51.473037 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.473016    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-46b60-kube-rbac-proxy-sar-config\""
Sep 16 13:05:51.473139 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.473052    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-46b60-serving-cert\""
Sep 16 13:05:51.477079 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.477037    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"]
Sep 16 13:05:51.598884 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.598844    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9158473-ed48-438f-b364-af229a7f6d19-openshift-service-ca-bundle\") pod \"sequence-graph-46b60-5cbdd9df9d-7d2vz\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") " pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:51.599050 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.598895    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9158473-ed48-438f-b364-af229a7f6d19-proxy-tls\") pod \"sequence-graph-46b60-5cbdd9df9d-7d2vz\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") " pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:51.699691 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.699654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9158473-ed48-438f-b364-af229a7f6d19-openshift-service-ca-bundle\") pod \"sequence-graph-46b60-5cbdd9df9d-7d2vz\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") " pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:51.699886 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.699701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9158473-ed48-438f-b364-af229a7f6d19-proxy-tls\") pod \"sequence-graph-46b60-5cbdd9df9d-7d2vz\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") " pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:51.700453 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.700432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9158473-ed48-438f-b364-af229a7f6d19-openshift-service-ca-bundle\") pod \"sequence-graph-46b60-5cbdd9df9d-7d2vz\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") " pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:51.702164 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.702140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9158473-ed48-438f-b364-af229a7f6d19-proxy-tls\") pod \"sequence-graph-46b60-5cbdd9df9d-7d2vz\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") " pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:51.784284 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:51.784177    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:52.052569 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:52.052546    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"]
Sep 16 13:05:52.055134 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:05:52.055107    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9158473_ed48_438f_b364_af229a7f6d19.slice/crio-ca757bff3d673dbb7e9a6e61530077d695ca78b79faec04a6f8966d786e05fbd WatchSource:0}: Error finding container ca757bff3d673dbb7e9a6e61530077d695ca78b79faec04a6f8966d786e05fbd: Status 404 returned error can't find the container with id ca757bff3d673dbb7e9a6e61530077d695ca78b79faec04a6f8966d786e05fbd
Sep 16 13:05:52.090146 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:52.090118    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" event={"ID":"a9158473-ed48-438f-b364-af229a7f6d19","Type":"ContainerStarted","Data":"ca757bff3d673dbb7e9a6e61530077d695ca78b79faec04a6f8966d786e05fbd"}
Sep 16 13:05:53.096397 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:53.096360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" event={"ID":"a9158473-ed48-438f-b364-af229a7f6d19","Type":"ContainerStarted","Data":"e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a"}
Sep 16 13:05:53.096870 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:53.096477    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:05:53.113331 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:53.113280    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podStartSLOduration=2.113264615 podStartE2EDuration="2.113264615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:53.112247152 +0000 UTC m=+1079.573875849" watchObservedRunningTime="2026-09-16 13:05:53.113264615 +0000 UTC m=+1079.574893353"
Sep 16 13:05:59.105566 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:05:59.105530    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:13:06.309928 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.309893    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"]
Sep 16 13:13:06.310467 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.310111    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a" containerID="cri-o://93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314" gracePeriod=30
Sep 16 13:13:06.505180 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.505139    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"]
Sep 16 13:13:06.505421 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.505386    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" containerID="cri-o://81f8d335999e7ba557b606e5ee698d2b0e61343f9f85fd545460a847de005204" gracePeriod=30
Sep 16 13:13:06.505555 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.505417    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kube-rbac-proxy" containerID="cri-o://0674ff8f45ce08926aefbe54ff7daf40040319d31655bf3cecfda722f94e7179" gracePeriod=30
Sep 16 13:13:06.570685 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.570609    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"]
Sep 16 13:13:06.573968 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.573952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.576499 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.576475    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bfe0b-predictor-serving-cert\""
Sep 16 13:13:06.576605 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.576499    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bfe0b-kube-rbac-proxy-sar-config\""
Sep 16 13:13:06.583304 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.583281    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"]
Sep 16 13:13:06.635141 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.635114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jkf\" (UniqueName: \"kubernetes.io/projected/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-api-access-28jkf\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.635269 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.635156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.635269 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.635192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a125a08-cccc-4799-aa16-1e3eb736b7f3-proxy-tls\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.701805 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.701768    2558 generic.go:356] "Generic (PLEG): container finished" podID="9065332b-0aad-41ac-ac58-5924cfea2119" containerID="0674ff8f45ce08926aefbe54ff7daf40040319d31655bf3cecfda722f94e7179" exitCode=2
Sep 16 13:13:06.701997 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.701854    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" event={"ID":"9065332b-0aad-41ac-ac58-5924cfea2119","Type":"ContainerDied","Data":"0674ff8f45ce08926aefbe54ff7daf40040319d31655bf3cecfda722f94e7179"}
Sep 16 13:13:06.736319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.736284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.736465 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.736333    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a125a08-cccc-4799-aa16-1e3eb736b7f3-proxy-tls\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.736465 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.736376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jkf\" (UniqueName: \"kubernetes.io/projected/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-api-access-28jkf\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.736949 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.736930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.738730 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.738711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a125a08-cccc-4799-aa16-1e3eb736b7f3-proxy-tls\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.744693 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.744672    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jkf\" (UniqueName: \"kubernetes.io/projected/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-api-access-28jkf\") pod \"error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") " pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:06.883549 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:06.883463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:07.155107 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:07.155082    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"]
Sep 16 13:13:07.157954 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:13:07.157928    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a125a08_cccc_4799_aa16_1e3eb736b7f3.slice/crio-2a7c4630e2c8b500fb77b1094a1f09c60bf283468a44d821723b0a1de0950834 WatchSource:0}: Error finding container 2a7c4630e2c8b500fb77b1094a1f09c60bf283468a44d821723b0a1de0950834: Status 404 returned error can't find the container with id 2a7c4630e2c8b500fb77b1094a1f09c60bf283468a44d821723b0a1de0950834
Sep 16 13:13:07.708439 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:07.708407    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" event={"ID":"3a125a08-cccc-4799-aa16-1e3eb736b7f3","Type":"ContainerStarted","Data":"1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef"}
Sep 16 13:13:07.708871 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:07.708475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" event={"ID":"3a125a08-cccc-4799-aa16-1e3eb736b7f3","Type":"ContainerStarted","Data":"a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15"}
Sep 16 13:13:07.708871 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:07.708505    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" event={"ID":"3a125a08-cccc-4799-aa16-1e3eb736b7f3","Type":"ContainerStarted","Data":"2a7c4630e2c8b500fb77b1094a1f09c60bf283468a44d821723b0a1de0950834"}
Sep 16 13:13:07.708871 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:07.708539    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:07.727167 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:07.727121    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podStartSLOduration=1.7271050030000001 podStartE2EDuration="1.727105003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:07.72596206 +0000 UTC m=+1514.187590755" watchObservedRunningTime="2026-09-16 13:13:07.727105003 +0000 UTC m=+1514.188733767"
Sep 16 13:13:08.712579 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:08.712545    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:08.713988 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:08.713959    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 13:13:08.878165 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:08.878125    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:13:09.635185 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.635145    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 13:13:09.641163 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.641123    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 13:13:09.718572 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.718538    2558 generic.go:356] "Generic (PLEG): container finished" podID="9065332b-0aad-41ac-ac58-5924cfea2119" containerID="81f8d335999e7ba557b606e5ee698d2b0e61343f9f85fd545460a847de005204" exitCode=0
Sep 16 13:13:09.719030 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.718604    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" event={"ID":"9065332b-0aad-41ac-ac58-5924cfea2119","Type":"ContainerDied","Data":"81f8d335999e7ba557b606e5ee698d2b0e61343f9f85fd545460a847de005204"}
Sep 16 13:13:09.719030 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.718913    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 13:13:09.750593 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.750562    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:13:09.862182 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.862097    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82chl\" (UniqueName: \"kubernetes.io/projected/9065332b-0aad-41ac-ac58-5924cfea2119-kube-api-access-82chl\") pod \"9065332b-0aad-41ac-ac58-5924cfea2119\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") "
Sep 16 13:13:09.862182 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.862170    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9065332b-0aad-41ac-ac58-5924cfea2119-kube-rbac-proxy-sar-config\") pod \"9065332b-0aad-41ac-ac58-5924cfea2119\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") "
Sep 16 13:13:09.862414 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.862199    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9065332b-0aad-41ac-ac58-5924cfea2119-proxy-tls\") pod \"9065332b-0aad-41ac-ac58-5924cfea2119\" (UID: \"9065332b-0aad-41ac-ac58-5924cfea2119\") "
Sep 16 13:13:09.862580 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.862546    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9065332b-0aad-41ac-ac58-5924cfea2119-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9065332b-0aad-41ac-ac58-5924cfea2119" (UID: "9065332b-0aad-41ac-ac58-5924cfea2119"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:13:09.864245 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.864222    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9065332b-0aad-41ac-ac58-5924cfea2119-kube-api-access-82chl" (OuterVolumeSpecName: "kube-api-access-82chl") pod "9065332b-0aad-41ac-ac58-5924cfea2119" (UID: "9065332b-0aad-41ac-ac58-5924cfea2119"). InnerVolumeSpecName "kube-api-access-82chl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:13:09.864327 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.864237    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9065332b-0aad-41ac-ac58-5924cfea2119-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9065332b-0aad-41ac-ac58-5924cfea2119" (UID: "9065332b-0aad-41ac-ac58-5924cfea2119"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:13:09.962986 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.962951    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9065332b-0aad-41ac-ac58-5924cfea2119-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:13:09.962986 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.962980    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82chl\" (UniqueName: \"kubernetes.io/projected/9065332b-0aad-41ac-ac58-5924cfea2119-kube-api-access-82chl\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:13:09.962986 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:09.962989    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9065332b-0aad-41ac-ac58-5924cfea2119-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:13:10.723601 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:10.723552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm" event={"ID":"9065332b-0aad-41ac-ac58-5924cfea2119","Type":"ContainerDied","Data":"ec30979f16a5dafad84f689da483790983a4aeaf2be383e9c546c47f54903a16"}
Sep 16 13:13:10.723601 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:10.723599    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"
Sep 16 13:13:10.724063 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:10.723660    2558 scope.go:117] "RemoveContainer" containerID="0674ff8f45ce08926aefbe54ff7daf40040319d31655bf3cecfda722f94e7179"
Sep 16 13:13:10.731669 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:10.731652    2558 scope.go:117] "RemoveContainer" containerID="81f8d335999e7ba557b606e5ee698d2b0e61343f9f85fd545460a847de005204"
Sep 16 13:13:10.744006 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:10.743984    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"]
Sep 16 13:13:10.746371 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:10.746348    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03a3a-predictor-864f8dfd-tx9rm"]
Sep 16 13:13:12.041708 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:12.041673    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" path="/var/lib/kubelet/pods/9065332b-0aad-41ac-ac58-5924cfea2119/volumes"
Sep 16 13:13:13.878140 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:13.878098    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:13:14.723599 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:14.723570    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:13:14.723987 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:14.723960    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 13:13:18.878340 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:18.878295    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:13:18.878753 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:18.878411    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:13:23.878543 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:23.878511    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:13:24.724898 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:24.724859    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 13:13:28.878306 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:28.878268    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:13:33.878691 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:33.878652    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:13:34.724901 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:34.724861    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 13:13:36.461311 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.461287    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:13:36.584393 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.584313    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7013f247-3f5c-40d5-a59b-e2ed07222a95-proxy-tls\") pod \"7013f247-3f5c-40d5-a59b-e2ed07222a95\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") "
Sep 16 13:13:36.584393 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.584365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7013f247-3f5c-40d5-a59b-e2ed07222a95-openshift-service-ca-bundle\") pod \"7013f247-3f5c-40d5-a59b-e2ed07222a95\" (UID: \"7013f247-3f5c-40d5-a59b-e2ed07222a95\") "
Sep 16 13:13:36.584762 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.584732    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7013f247-3f5c-40d5-a59b-e2ed07222a95-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7013f247-3f5c-40d5-a59b-e2ed07222a95" (UID: "7013f247-3f5c-40d5-a59b-e2ed07222a95"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:13:36.586416 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.586386    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7013f247-3f5c-40d5-a59b-e2ed07222a95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7013f247-3f5c-40d5-a59b-e2ed07222a95" (UID: "7013f247-3f5c-40d5-a59b-e2ed07222a95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:13:36.685083 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.685042    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7013f247-3f5c-40d5-a59b-e2ed07222a95-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:13:36.685083 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.685076    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7013f247-3f5c-40d5-a59b-e2ed07222a95-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:13:36.821856 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.821804    2558 generic.go:356] "Generic (PLEG): container finished" podID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerID="93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314" exitCode=0
Sep 16 13:13:36.822016 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.821862    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"
Sep 16 13:13:36.822016 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.821905    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" event={"ID":"7013f247-3f5c-40d5-a59b-e2ed07222a95","Type":"ContainerDied","Data":"93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314"}
Sep 16 13:13:36.822016 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.821952    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg" event={"ID":"7013f247-3f5c-40d5-a59b-e2ed07222a95","Type":"ContainerDied","Data":"b4855b9cfaeca65174418ac76282cd497c004143d47be491efa7c9c60530eb17"}
Sep 16 13:13:36.822016 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.821975    2558 scope.go:117] "RemoveContainer" containerID="93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314"
Sep 16 13:13:36.831213 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.831192    2558 scope.go:117] "RemoveContainer" containerID="93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314"
Sep 16 13:13:36.831494 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:13:36.831475    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314\": container with ID starting with 93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314 not found: ID does not exist" containerID="93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314"
Sep 16 13:13:36.831580 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.831503    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314"} err="failed to get container status \"93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314\": rpc error: code = NotFound desc = could not find container \"93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314\": container with ID starting with 93aa890a5ff6b7422f888541880cefd41e28883d35fcce54bb3cea6ae8f64314 not found: ID does not exist"
Sep 16 13:13:36.843702 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.843641    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"]
Sep 16 13:13:36.847040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:36.847022    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-03a3a-5ff944788f-79jqg"]
Sep 16 13:13:38.041805 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:38.041761    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" path="/var/lib/kubelet/pods/7013f247-3f5c-40d5-a59b-e2ed07222a95/volumes"
Sep 16 13:13:44.724802 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:44.724768    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 13:13:54.724961 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:13:54.724931    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:14:06.221523 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.221486    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"]
Sep 16 13:14:06.222006 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.221764    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60" containerID="cri-o://e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a" gracePeriod=30
Sep 16 13:14:06.340209 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.340170    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"]
Sep 16 13:14:06.340518 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.340475    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container" containerID="cri-o://bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35" gracePeriod=30
Sep 16 13:14:06.340587 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.340499    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kube-rbac-proxy" containerID="cri-o://7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc" gracePeriod=30
Sep 16 13:14:06.494000 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.493913    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"]
Sep 16 13:14:06.494226 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494211    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a"
Sep 16 13:14:06.494226 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494226    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a"
Sep 16 13:14:06.494355 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494239    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kube-rbac-proxy"
Sep 16 13:14:06.494355 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494244    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kube-rbac-proxy"
Sep 16 13:14:06.494355 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494263    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container"
Sep 16 13:14:06.494355 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494268    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container"
Sep 16 13:14:06.494355 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494315    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7013f247-3f5c-40d5-a59b-e2ed07222a95" containerName="switch-graph-03a3a"
Sep 16 13:14:06.494355 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494324    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kserve-container"
Sep 16 13:14:06.494355 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.494330    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9065332b-0aad-41ac-ac58-5924cfea2119" containerName="kube-rbac-proxy"
Sep 16 13:14:06.497234 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.497212    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.499803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.499779    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aa957-predictor-serving-cert\""
Sep 16 13:14:06.499803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.499785    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aa957-kube-rbac-proxy-sar-config\""
Sep 16 13:14:06.507353 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.507316    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"]
Sep 16 13:14:06.508664 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.508641    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"]
Sep 16 13:14:06.511754 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.511735    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.514103 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.514085    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-bfe0b-serving-cert\""
Sep 16 13:14:06.514215 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.514108    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-bfe0b-kube-rbac-proxy-sar-config\""
Sep 16 13:14:06.522714 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.522685    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"]
Sep 16 13:14:06.614587 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.614552    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zv87\" (UniqueName: \"kubernetes.io/projected/5b866478-1a7b-4b0d-8780-728008d47114-kube-api-access-6zv87\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.614587 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.614587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-openshift-service-ca-bundle\") pod \"ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") " pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.614794 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.614626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b866478-1a7b-4b0d-8780-728008d47114-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.614794 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.614679    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-proxy-tls\") pod \"ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") " pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.614794 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.614720    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.715907 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.715869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.716102 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.715919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zv87\" (UniqueName: \"kubernetes.io/projected/5b866478-1a7b-4b0d-8780-728008d47114-kube-api-access-6zv87\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.716102 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.715942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-openshift-service-ca-bundle\") pod \"ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") " pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.716102 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.715974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b866478-1a7b-4b0d-8780-728008d47114-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.716102 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.716000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-proxy-tls\") pod \"ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") " pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.716102 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:06.716027    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-aa957-predictor-serving-cert: secret "error-404-isvc-aa957-predictor-serving-cert" not found
Sep 16 13:14:06.716102 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:06.716095    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls podName:5b866478-1a7b-4b0d-8780-728008d47114 nodeName:}" failed. No retries permitted until 2026-09-16 13:14:07.21607516 +0000 UTC m=+1573.677703859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls") pod "error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" (UID: "5b866478-1a7b-4b0d-8780-728008d47114") : secret "error-404-isvc-aa957-predictor-serving-cert" not found
Sep 16 13:14:06.716701 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.716677    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-openshift-service-ca-bundle\") pod \"ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") " pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.716786 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.716722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b866478-1a7b-4b0d-8780-728008d47114-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.718523 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.718503    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-proxy-tls\") pod \"ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") " pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.724437 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.724416    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zv87\" (UniqueName: \"kubernetes.io/projected/5b866478-1a7b-4b0d-8780-728008d47114-kube-api-access-6zv87\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:06.820742 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.820654    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:06.938423 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.938388    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerID="7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc" exitCode=2
Sep 16 13:14:06.938569 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:06.938437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" event={"ID":"fa4861b0-6e22-42b0-aa77-99e87d3a4354","Type":"ContainerDied","Data":"7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc"}
Sep 16 13:14:07.089427 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.089215    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"]
Sep 16 13:14:07.091931 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:14:07.091907    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c61b4a7_5be5_482c_ac95_35ab7d0e0399.slice/crio-ad93fe8b0560189df9999d33c7e81254758230da8c1933d1fe5e1758ce0dc30f WatchSource:0}: Error finding container ad93fe8b0560189df9999d33c7e81254758230da8c1933d1fe5e1758ce0dc30f: Status 404 returned error can't find the container with id ad93fe8b0560189df9999d33c7e81254758230da8c1933d1fe5e1758ce0dc30f
Sep 16 13:14:07.220992 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.220954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:07.223333 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.223310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls\") pod \"error-404-isvc-aa957-predictor-85ccdd9d88-88xfx\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") " pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:07.408832 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.408782    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:07.679769 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.679698    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"]
Sep 16 13:14:07.682647 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:14:07.682619    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b866478_1a7b_4b0d_8780_728008d47114.slice/crio-054d3a8bc66ba4a9f2ed45eda6856bdc503226709ad3f49537cdaa1a47bc1b06 WatchSource:0}: Error finding container 054d3a8bc66ba4a9f2ed45eda6856bdc503226709ad3f49537cdaa1a47bc1b06: Status 404 returned error can't find the container with id 054d3a8bc66ba4a9f2ed45eda6856bdc503226709ad3f49537cdaa1a47bc1b06
Sep 16 13:14:07.943821 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.943703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" event={"ID":"5b866478-1a7b-4b0d-8780-728008d47114","Type":"ContainerStarted","Data":"08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f"}
Sep 16 13:14:07.943821 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.943774    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" event={"ID":"5b866478-1a7b-4b0d-8780-728008d47114","Type":"ContainerStarted","Data":"3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f"}
Sep 16 13:14:07.943821 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.943800    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:07.944133 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.943926    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:07.944133 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.943979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" event={"ID":"5b866478-1a7b-4b0d-8780-728008d47114","Type":"ContainerStarted","Data":"054d3a8bc66ba4a9f2ed45eda6856bdc503226709ad3f49537cdaa1a47bc1b06"}
Sep 16 13:14:07.945285 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.945251    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 13:14:07.945416 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.945341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" event={"ID":"5c61b4a7-5be5-482c-ac95-35ab7d0e0399","Type":"ContainerStarted","Data":"157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b"}
Sep 16 13:14:07.945416 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.945374    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" event={"ID":"5c61b4a7-5be5-482c-ac95-35ab7d0e0399","Type":"ContainerStarted","Data":"ad93fe8b0560189df9999d33c7e81254758230da8c1933d1fe5e1758ce0dc30f"}
Sep 16 13:14:07.945533 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.945520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:07.961576 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.961523    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podStartSLOduration=1.961508374 podStartE2EDuration="1.961508374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:07.960970489 +0000 UTC m=+1574.422599195" watchObservedRunningTime="2026-09-16 13:14:07.961508374 +0000 UTC m=+1574.423137069"
Sep 16 13:14:07.976122 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:07.976070    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podStartSLOduration=1.976051273 podStartE2EDuration="1.976051273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:07.975201122 +0000 UTC m=+1574.436829818" watchObservedRunningTime="2026-09-16 13:14:07.976051273 +0000 UTC m=+1574.437679968"
Sep 16 13:14:08.949714 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:08.949675    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 13:14:09.104255 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.104215    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:09.690122 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.690096    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:14:09.741054 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.740964    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b0-6e22-42b0-aa77-99e87d3a4354-proxy-tls\") pod \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") "
Sep 16 13:14:09.741054 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.741033    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hccsc\" (UniqueName: \"kubernetes.io/projected/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-api-access-hccsc\") pod \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") "
Sep 16 13:14:09.741234 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.741105    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-rbac-proxy-sar-config\") pod \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\" (UID: \"fa4861b0-6e22-42b0-aa77-99e87d3a4354\") "
Sep 16 13:14:09.741445 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.741418    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa4861b0-6e22-42b0-aa77-99e87d3a4354" (UID: "fa4861b0-6e22-42b0-aa77-99e87d3a4354"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:09.743104 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.743081    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-api-access-hccsc" (OuterVolumeSpecName: "kube-api-access-hccsc") pod "fa4861b0-6e22-42b0-aa77-99e87d3a4354" (UID: "fa4861b0-6e22-42b0-aa77-99e87d3a4354"). InnerVolumeSpecName "kube-api-access-hccsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:09.743190 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.743176    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4861b0-6e22-42b0-aa77-99e87d3a4354-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa4861b0-6e22-42b0-aa77-99e87d3a4354" (UID: "fa4861b0-6e22-42b0-aa77-99e87d3a4354"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:14:09.841756 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.841716    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hccsc\" (UniqueName: \"kubernetes.io/projected/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-api-access-hccsc\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:09.841756 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.841749    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b0-6e22-42b0-aa77-99e87d3a4354-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:09.841756 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.841760    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b0-6e22-42b0-aa77-99e87d3a4354-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:09.954691 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.954659    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerID="bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35" exitCode=0
Sep 16 13:14:09.955129 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.954713    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"
Sep 16 13:14:09.955129 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.954755    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" event={"ID":"fa4861b0-6e22-42b0-aa77-99e87d3a4354","Type":"ContainerDied","Data":"bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35"}
Sep 16 13:14:09.955129 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.954844    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk" event={"ID":"fa4861b0-6e22-42b0-aa77-99e87d3a4354","Type":"ContainerDied","Data":"2ebab2874c5182da55d5484aa32b841806bcff63459944d7b518c214127c074e"}
Sep 16 13:14:09.955129 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.954891    2558 scope.go:117] "RemoveContainer" containerID="7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc"
Sep 16 13:14:09.963678 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.963660    2558 scope.go:117] "RemoveContainer" containerID="bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35"
Sep 16 13:14:09.971377 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.971359    2558 scope.go:117] "RemoveContainer" containerID="7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc"
Sep 16 13:14:09.971658 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:09.971638    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc\": container with ID starting with 7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc not found: ID does not exist" containerID="7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc"
Sep 16 13:14:09.971704 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.971665    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc"} err="failed to get container status \"7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc\": rpc error: code = NotFound desc = could not find container \"7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc\": container with ID starting with 7cd41d150589f89a7049c16173a234716e6f2a5de79af9520fe1832d2ad21bcc not found: ID does not exist"
Sep 16 13:14:09.971704 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.971682    2558 scope.go:117] "RemoveContainer" containerID="bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35"
Sep 16 13:14:09.971925 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:09.971907    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35\": container with ID starting with bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35 not found: ID does not exist" containerID="bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35"
Sep 16 13:14:09.971970 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.971931    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35"} err="failed to get container status \"bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35\": rpc error: code = NotFound desc = could not find container \"bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35\": container with ID starting with bbce963b006e4bddef8c1348bcc740896f4f5911637727085e11b94c1064aa35 not found: ID does not exist"
Sep 16 13:14:09.975310 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.975283    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"]
Sep 16 13:14:09.977170 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:09.977149    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-46b60-predictor-786fd95898-n46zk"]
Sep 16 13:14:10.041936 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:10.041851    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" path="/var/lib/kubelet/pods/fa4861b0-6e22-42b0-aa77-99e87d3a4354/volumes"
Sep 16 13:14:13.954727 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:13.954693    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:13.955205 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:13.955171    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:13.955304 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:13.955272    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 13:14:14.104136 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:14.104099    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:16.583638 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.583556    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"]
Sep 16 13:14:16.584039 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.583767    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b" containerID="cri-o://157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b" gracePeriod=30
Sep 16 13:14:16.696946 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.696912    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"]
Sep 16 13:14:16.697178 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.697151    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container" containerID="cri-o://a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15" gracePeriod=30
Sep 16 13:14:16.697266 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.697175    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kube-rbac-proxy" containerID="cri-o://1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef" gracePeriod=30
Sep 16 13:14:16.860646 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.860559    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"]
Sep 16 13:14:16.860933 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.860919    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kube-rbac-proxy"
Sep 16 13:14:16.860980 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.860936    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kube-rbac-proxy"
Sep 16 13:14:16.860980 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.860955    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container"
Sep 16 13:14:16.860980 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.860963    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container"
Sep 16 13:14:16.861095 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.861016    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kube-rbac-proxy"
Sep 16 13:14:16.861095 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.861028    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4861b0-6e22-42b0-aa77-99e87d3a4354" containerName="kserve-container"
Sep 16 13:14:16.863889 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.863871    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:16.866416 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.866397    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d30d6-predictor-serving-cert\""
Sep 16 13:14:16.866507 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.866401    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d30d6-kube-rbac-proxy-sar-config\""
Sep 16 13:14:16.873854 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.873834    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"]
Sep 16 13:14:16.982682 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.982652    2558 generic.go:356] "Generic (PLEG): container finished" podID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerID="1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef" exitCode=2
Sep 16 13:14:16.982887 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:16.982726    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" event={"ID":"3a125a08-cccc-4799-aa16-1e3eb736b7f3","Type":"ContainerDied","Data":"1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef"}
Sep 16 13:14:17.004083 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.004055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a0ab6-e0a0-4d7f-b03a-c3484edce952-proxy-tls\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.004167 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.004091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz2s\" (UniqueName: \"kubernetes.io/projected/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-api-access-npz2s\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.004167 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.004114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.104859 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.104829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a0ab6-e0a0-4d7f-b03a-c3484edce952-proxy-tls\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.105021 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.104869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz2s\" (UniqueName: \"kubernetes.io/projected/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-api-access-npz2s\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.105021 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.104897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.105640 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.105614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.107317 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.107298    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a0ab6-e0a0-4d7f-b03a-c3484edce952-proxy-tls\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.117570 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.117517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz2s\" (UniqueName: \"kubernetes.io/projected/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-api-access-npz2s\") pod \"error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") " pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.173729 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.173689    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:17.443099 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.443074    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"]
Sep 16 13:14:17.445496 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:14:17.445468    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141a0ab6_e0a0_4d7f_b03a_c3484edce952.slice/crio-efca864a8e92764e1b82572454219a1da0c67336f1785a3d467782876ec5f5e0 WatchSource:0}: Error finding container efca864a8e92764e1b82572454219a1da0c67336f1785a3d467782876ec5f5e0: Status 404 returned error can't find the container with id efca864a8e92764e1b82572454219a1da0c67336f1785a3d467782876ec5f5e0
Sep 16 13:14:17.988747 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.988712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" event={"ID":"141a0ab6-e0a0-4d7f-b03a-c3484edce952","Type":"ContainerStarted","Data":"d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1"}
Sep 16 13:14:17.989179 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.988768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" event={"ID":"141a0ab6-e0a0-4d7f-b03a-c3484edce952","Type":"ContainerStarted","Data":"e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6"}
Sep 16 13:14:17.989179 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.988788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" event={"ID":"141a0ab6-e0a0-4d7f-b03a-c3484edce952","Type":"ContainerStarted","Data":"efca864a8e92764e1b82572454219a1da0c67336f1785a3d467782876ec5f5e0"}
Sep 16 13:14:17.989179 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:17.988851    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:18.007887 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:18.007839    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podStartSLOduration=2.007823631 podStartE2EDuration="2.007823631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:18.005482631 +0000 UTC m=+1584.467111331" watchObservedRunningTime="2026-09-16 13:14:18.007823631 +0000 UTC m=+1584.469452317"
Sep 16 13:14:18.955138 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:18.955092    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:18.992625 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:18.992598    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:18.994203 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:18.994174    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 13:14:19.104298 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:19.104263    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:19.104451 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:19.104360    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:14:19.719600 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:19.719554    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 13:14:19.996990 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:19.996907    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 13:14:20.341770 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.341748    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:14:20.435361 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.435325    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28jkf\" (UniqueName: \"kubernetes.io/projected/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-api-access-28jkf\") pod \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") "
Sep 16 13:14:20.435559 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.435380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-rbac-proxy-sar-config\") pod \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") "
Sep 16 13:14:20.435559 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.435455    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a125a08-cccc-4799-aa16-1e3eb736b7f3-proxy-tls\") pod \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\" (UID: \"3a125a08-cccc-4799-aa16-1e3eb736b7f3\") "
Sep 16 13:14:20.435766 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.435739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a125a08-cccc-4799-aa16-1e3eb736b7f3" (UID: "3a125a08-cccc-4799-aa16-1e3eb736b7f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:20.437443 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.437412    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a125a08-cccc-4799-aa16-1e3eb736b7f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a125a08-cccc-4799-aa16-1e3eb736b7f3" (UID: "3a125a08-cccc-4799-aa16-1e3eb736b7f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:14:20.437443 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.437413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-api-access-28jkf" (OuterVolumeSpecName: "kube-api-access-28jkf") pod "3a125a08-cccc-4799-aa16-1e3eb736b7f3" (UID: "3a125a08-cccc-4799-aa16-1e3eb736b7f3"). InnerVolumeSpecName "kube-api-access-28jkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:20.536878 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.536781    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jkf\" (UniqueName: \"kubernetes.io/projected/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-api-access-28jkf\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:20.536878 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.536805    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a125a08-cccc-4799-aa16-1e3eb736b7f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:20.536878 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:20.536828    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a125a08-cccc-4799-aa16-1e3eb736b7f3-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:21.002375 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.002341    2558 generic.go:356] "Generic (PLEG): container finished" podID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerID="a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15" exitCode=0
Sep 16 13:14:21.002784 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.002386    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"
Sep 16 13:14:21.002784 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.002424    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" event={"ID":"3a125a08-cccc-4799-aa16-1e3eb736b7f3","Type":"ContainerDied","Data":"a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15"}
Sep 16 13:14:21.002784 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.002487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs" event={"ID":"3a125a08-cccc-4799-aa16-1e3eb736b7f3","Type":"ContainerDied","Data":"2a7c4630e2c8b500fb77b1094a1f09c60bf283468a44d821723b0a1de0950834"}
Sep 16 13:14:21.002784 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.002515    2558 scope.go:117] "RemoveContainer" containerID="1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef"
Sep 16 13:14:21.010876 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.010857    2558 scope.go:117] "RemoveContainer" containerID="a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15"
Sep 16 13:14:21.018119 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.018099    2558 scope.go:117] "RemoveContainer" containerID="1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef"
Sep 16 13:14:21.018371 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:21.018353    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef\": container with ID starting with 1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef not found: ID does not exist" containerID="1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef"
Sep 16 13:14:21.018425 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.018379    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef"} err="failed to get container status \"1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef\": rpc error: code = NotFound desc = could not find container \"1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef\": container with ID starting with 1b899b67fd36d7a7ea26ff7262643c8dd93698ea360533ef4a1976bf4d095aef not found: ID does not exist"
Sep 16 13:14:21.018425 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.018396    2558 scope.go:117] "RemoveContainer" containerID="a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15"
Sep 16 13:14:21.018630 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:21.018613    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15\": container with ID starting with a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15 not found: ID does not exist" containerID="a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15"
Sep 16 13:14:21.018677 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.018648    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15"} err="failed to get container status \"a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15\": rpc error: code = NotFound desc = could not find container \"a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15\": container with ID starting with a7d8aa1390f12207512c53e70e91f148a9771025887d93b2621f30f4fc97ac15 not found: ID does not exist"
Sep 16 13:14:21.022895 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.022872    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"]
Sep 16 13:14:21.026051 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:21.026032    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bfe0b-predictor-c5c5ddc64-z8xqs"]
Sep 16 13:14:22.041693 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:22.041660    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" path="/var/lib/kubelet/pods/3a125a08-cccc-4799-aa16-1e3eb736b7f3/volumes"
Sep 16 13:14:23.954318 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:23.954268    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:23.955397 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:23.955367    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 13:14:24.104375 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:24.104335    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:25.001972 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:25.001944    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:14:25.002465 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:25.002435    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 13:14:28.953931 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:28.953887    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:28.954308 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:28.954011    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:29.104492 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:29.104458    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:33.954301 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:33.954259    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:33.955428 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:33.955406    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 13:14:34.103845 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:34.103785    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:35.003202 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:35.003160    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 13:14:36.378461 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:36.378438    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:14:36.574076 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:36.573981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9158473-ed48-438f-b364-af229a7f6d19-proxy-tls\") pod \"a9158473-ed48-438f-b364-af229a7f6d19\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") "
Sep 16 13:14:36.574076 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:36.574066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9158473-ed48-438f-b364-af229a7f6d19-openshift-service-ca-bundle\") pod \"a9158473-ed48-438f-b364-af229a7f6d19\" (UID: \"a9158473-ed48-438f-b364-af229a7f6d19\") "
Sep 16 13:14:36.574445 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:36.574420    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9158473-ed48-438f-b364-af229a7f6d19-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a9158473-ed48-438f-b364-af229a7f6d19" (UID: "a9158473-ed48-438f-b364-af229a7f6d19"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:36.576219 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:36.576190    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9158473-ed48-438f-b364-af229a7f6d19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9158473-ed48-438f-b364-af229a7f6d19" (UID: "a9158473-ed48-438f-b364-af229a7f6d19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:14:36.674676 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:36.674637    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9158473-ed48-438f-b364-af229a7f6d19-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:36.674676 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:36.674669    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9158473-ed48-438f-b364-af229a7f6d19-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:37.068458 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.068424    2558 generic.go:356] "Generic (PLEG): container finished" podID="a9158473-ed48-438f-b364-af229a7f6d19" containerID="e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a" exitCode=0
Sep 16 13:14:37.068659 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.068466    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"
Sep 16 13:14:37.068659 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.068480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" event={"ID":"a9158473-ed48-438f-b364-af229a7f6d19","Type":"ContainerDied","Data":"e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a"}
Sep 16 13:14:37.068659 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.068527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz" event={"ID":"a9158473-ed48-438f-b364-af229a7f6d19","Type":"ContainerDied","Data":"ca757bff3d673dbb7e9a6e61530077d695ca78b79faec04a6f8966d786e05fbd"}
Sep 16 13:14:37.068659 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.068556    2558 scope.go:117] "RemoveContainer" containerID="e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a"
Sep 16 13:14:37.076891 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.076871    2558 scope.go:117] "RemoveContainer" containerID="e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a"
Sep 16 13:14:37.077150 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:37.077130    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a\": container with ID starting with e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a not found: ID does not exist" containerID="e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a"
Sep 16 13:14:37.077193 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.077159    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a"} err="failed to get container status \"e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a\": rpc error: code = NotFound desc = could not find container \"e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a\": container with ID starting with e7f429dabb6bf0fe53a7404963c7a8df992d81d37034a9378f60c353b129504a not found: ID does not exist"
Sep 16 13:14:37.088890 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.088866    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"]
Sep 16 13:14:37.092058 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:37.092041    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46b60-5cbdd9df9d-7d2vz"]
Sep 16 13:14:38.041393 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:38.041360    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9158473-ed48-438f-b364-af229a7f6d19" path="/var/lib/kubelet/pods/a9158473-ed48-438f-b364-af229a7f6d19/volumes"
Sep 16 13:14:38.953777 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:38.953736    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:43.954331 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:43.954292    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:14:43.955526 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:43.955495    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 13:14:45.002619 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:45.002580    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 13:14:46.730407 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:46.730384    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:46.740418 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:46.740398    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-openshift-service-ca-bundle\") pod \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") "
Sep 16 13:14:46.740514 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:46.740435    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-proxy-tls\") pod \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\" (UID: \"5c61b4a7-5be5-482c-ac95-35ab7d0e0399\") "
Sep 16 13:14:46.740707 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:46.740687    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5c61b4a7-5be5-482c-ac95-35ab7d0e0399" (UID: "5c61b4a7-5be5-482c-ac95-35ab7d0e0399"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:46.742341 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:46.742317    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c61b4a7-5be5-482c-ac95-35ab7d0e0399" (UID: "5c61b4a7-5be5-482c-ac95-35ab7d0e0399"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:14:46.841000 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:46.840930    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:46.841000 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:46.840961    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c61b4a7-5be5-482c-ac95-35ab7d0e0399-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:14:47.115257 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.115175    2558 generic.go:356] "Generic (PLEG): container finished" podID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerID="157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b" exitCode=137
Sep 16 13:14:47.115257 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.115223    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"
Sep 16 13:14:47.115483 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.115266    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" event={"ID":"5c61b4a7-5be5-482c-ac95-35ab7d0e0399","Type":"ContainerDied","Data":"157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b"}
Sep 16 13:14:47.115483 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.115328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p" event={"ID":"5c61b4a7-5be5-482c-ac95-35ab7d0e0399","Type":"ContainerDied","Data":"ad93fe8b0560189df9999d33c7e81254758230da8c1933d1fe5e1758ce0dc30f"}
Sep 16 13:14:47.115483 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.115358    2558 scope.go:117] "RemoveContainer" containerID="157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b"
Sep 16 13:14:47.124116 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.123972    2558 scope.go:117] "RemoveContainer" containerID="157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b"
Sep 16 13:14:47.124311 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:14:47.124287    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b\": container with ID starting with 157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b not found: ID does not exist" containerID="157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b"
Sep 16 13:14:47.124383 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.124323    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b"} err="failed to get container status \"157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b\": rpc error: code = NotFound desc = could not find container \"157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b\": container with ID starting with 157543318d6b61a46e0c368c947091fe0c891e046a9b641c3b6a26d1af0d0b1b not found: ID does not exist"
Sep 16 13:14:47.160490 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.160466    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"]
Sep 16 13:14:47.170692 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:47.170673    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bfe0b-7bfd7d4fb7-hkq8p"]
Sep 16 13:14:48.042267 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:48.042230    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" path="/var/lib/kubelet/pods/5c61b4a7-5be5-482c-ac95-35ab7d0e0399/volumes"
Sep 16 13:14:53.956707 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:53.956674    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:14:55.003286 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:14:55.003245    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 13:15:05.003640 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:05.003610    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:15:06.431558 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431520    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"]
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431833    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431845    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431862    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kube-rbac-proxy"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431869    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kube-rbac-proxy"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431877    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431883    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431895    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431900    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container"
Sep 16 13:15:06.431944 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431943    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kube-rbac-proxy"
Sep 16 13:15:06.432258 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431952    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c61b4a7-5be5-482c-ac95-35ab7d0e0399" containerName="ensemble-graph-bfe0b"
Sep 16 13:15:06.432258 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431959    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9158473-ed48-438f-b364-af229a7f6d19" containerName="sequence-graph-46b60"
Sep 16 13:15:06.432258 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.431964    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a125a08-cccc-4799-aa16-1e3eb736b7f3" containerName="kserve-container"
Sep 16 13:15:06.434734 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.434717    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:06.437145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.437124    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-aa957-serving-cert\""
Sep 16 13:15:06.437270 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.437234    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-aa957-kube-rbac-proxy-sar-config\""
Sep 16 13:15:06.443455 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.443435    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"]
Sep 16 13:15:06.499890 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.499843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa02ad5-34fb-4042-9150-cc878e7b87ec-openshift-service-ca-bundle\") pod \"sequence-graph-aa957-55d5b78657-vbgkk\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") " pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:06.500048 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.499989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa02ad5-34fb-4042-9150-cc878e7b87ec-proxy-tls\") pod \"sequence-graph-aa957-55d5b78657-vbgkk\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") " pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:06.600973 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.600912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa02ad5-34fb-4042-9150-cc878e7b87ec-openshift-service-ca-bundle\") pod \"sequence-graph-aa957-55d5b78657-vbgkk\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") " pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:06.601169 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.601032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa02ad5-34fb-4042-9150-cc878e7b87ec-proxy-tls\") pod \"sequence-graph-aa957-55d5b78657-vbgkk\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") " pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:06.601595 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.601564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa02ad5-34fb-4042-9150-cc878e7b87ec-openshift-service-ca-bundle\") pod \"sequence-graph-aa957-55d5b78657-vbgkk\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") " pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:06.603390 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.603371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa02ad5-34fb-4042-9150-cc878e7b87ec-proxy-tls\") pod \"sequence-graph-aa957-55d5b78657-vbgkk\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") " pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:06.745464 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:06.745361    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:07.012603 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:07.012534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"]
Sep 16 13:15:07.015803 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:15:07.015776    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa02ad5_34fb_4042_9150_cc878e7b87ec.slice/crio-408ee73ca113cb80648bcea4bdbdfb4e56f39ce5d0ed88dfb643cf00f669fdfb WatchSource:0}: Error finding container 408ee73ca113cb80648bcea4bdbdfb4e56f39ce5d0ed88dfb643cf00f669fdfb: Status 404 returned error can't find the container with id 408ee73ca113cb80648bcea4bdbdfb4e56f39ce5d0ed88dfb643cf00f669fdfb
Sep 16 13:15:07.191504 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:07.191466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" event={"ID":"8fa02ad5-34fb-4042-9150-cc878e7b87ec","Type":"ContainerStarted","Data":"36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c"}
Sep 16 13:15:07.191674 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:07.191523    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" event={"ID":"8fa02ad5-34fb-4042-9150-cc878e7b87ec","Type":"ContainerStarted","Data":"408ee73ca113cb80648bcea4bdbdfb4e56f39ce5d0ed88dfb643cf00f669fdfb"}
Sep 16 13:15:07.191674 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:07.191560    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:07.206990 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:07.206925    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podStartSLOduration=1.2069072570000001 podStartE2EDuration="1.206907257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:07.206366556 +0000 UTC m=+1633.667995253" watchObservedRunningTime="2026-09-16 13:15:07.206907257 +0000 UTC m=+1633.668535954"
Sep 16 13:15:13.201301 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:13.201267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:16.517008 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.516971    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"]
Sep 16 13:15:16.517503 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.517215    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957" containerID="cri-o://36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c" gracePeriod=30
Sep 16 13:15:16.633989 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.633910    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"]
Sep 16 13:15:16.634191 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.634149    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container" containerID="cri-o://3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f" gracePeriod=30
Sep 16 13:15:16.634252 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.634182    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kube-rbac-proxy" containerID="cri-o://08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f" gracePeriod=30
Sep 16 13:15:16.815699 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.815666    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"]
Sep 16 13:15:16.819073 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.819048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.821490 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.821461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4c3c9-predictor-serving-cert\""
Sep 16 13:15:16.821576 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.821477    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4c3c9-kube-rbac-proxy-sar-config\""
Sep 16 13:15:16.824276 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.824257    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"]
Sep 16 13:15:16.827424 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.827409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:16.829947 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.829928    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d30d6-kube-rbac-proxy-sar-config\""
Sep 16 13:15:16.830049 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.829929    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d30d6-serving-cert\""
Sep 16 13:15:16.831742 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.831723    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"]
Sep 16 13:15:16.835715 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.835679    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"]
Sep 16 13:15:16.887083 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.886999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtqk\" (UniqueName: \"kubernetes.io/projected/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-api-access-zjtqk\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.887083 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.887056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.887273 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.887110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.887273 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.887139    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0823a3d-37c3-408d-aed8-5b315b6f1381-openshift-service-ca-bundle\") pod \"ensemble-graph-d30d6-6486bfbb6f-42c7p\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") " pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:16.887273 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.887173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls\") pod \"ensemble-graph-d30d6-6486bfbb6f-42c7p\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") " pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:16.988055 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.988014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtqk\" (UniqueName: \"kubernetes.io/projected/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-api-access-zjtqk\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.988226 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.988067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.988226 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.988139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.988226 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.988188    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0823a3d-37c3-408d-aed8-5b315b6f1381-openshift-service-ca-bundle\") pod \"ensemble-graph-d30d6-6486bfbb6f-42c7p\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") " pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:16.988226 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.988215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls\") pod \"ensemble-graph-d30d6-6486bfbb6f-42c7p\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") " pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:16.988416 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:15:16.988237    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-serving-cert: secret "error-404-isvc-4c3c9-predictor-serving-cert" not found
Sep 16 13:15:16.988416 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:15:16.988301    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls podName:019e4224-cfaa-40fa-9eca-65aaee18a3df nodeName:}" failed. No retries permitted until 2026-09-16 13:15:17.488280779 +0000 UTC m=+1643.949909472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls") pod "error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" (UID: "019e4224-cfaa-40fa-9eca-65aaee18a3df") : secret "error-404-isvc-4c3c9-predictor-serving-cert" not found
Sep 16 13:15:16.988416 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:15:16.988331    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-d30d6-serving-cert: secret "ensemble-graph-d30d6-serving-cert" not found
Sep 16 13:15:16.988416 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:15:16.988371    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls podName:a0823a3d-37c3-408d-aed8-5b315b6f1381 nodeName:}" failed. No retries permitted until 2026-09-16 13:15:17.488360956 +0000 UTC m=+1643.949989641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls") pod "ensemble-graph-d30d6-6486bfbb6f-42c7p" (UID: "a0823a3d-37c3-408d-aed8-5b315b6f1381") : secret "ensemble-graph-d30d6-serving-cert" not found
Sep 16 13:15:16.988740 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.988722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:16.988791 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.988775    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0823a3d-37c3-408d-aed8-5b315b6f1381-openshift-service-ca-bundle\") pod \"ensemble-graph-d30d6-6486bfbb6f-42c7p\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") " pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:16.996527 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:16.996507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtqk\" (UniqueName: \"kubernetes.io/projected/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-api-access-zjtqk\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:17.227123 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.227088    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b866478-1a7b-4b0d-8780-728008d47114" containerID="08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f" exitCode=2
Sep 16 13:15:17.227123 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.227127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" event={"ID":"5b866478-1a7b-4b0d-8780-728008d47114","Type":"ContainerDied","Data":"08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f"}
Sep 16 13:15:17.492569 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.492487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:17.492569 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.492527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls\") pod \"ensemble-graph-d30d6-6486bfbb6f-42c7p\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") " pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:17.494860 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.494825    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls\") pod \"error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") " pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:17.494996 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.494931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls\") pod \"ensemble-graph-d30d6-6486bfbb6f-42c7p\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") " pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:17.729373 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.729331    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:17.738555 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:17.738532    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:18.004607 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.004537    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"]
Sep 16 13:15:18.007654 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:15:18.007625    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019e4224_cfaa_40fa_9eca_65aaee18a3df.slice/crio-5b45b1aff7a0f246f5c2ff1bbc7bc4ac28c6fdd752633e3946e8f4976e34e40c WatchSource:0}: Error finding container 5b45b1aff7a0f246f5c2ff1bbc7bc4ac28c6fdd752633e3946e8f4976e34e40c: Status 404 returned error can't find the container with id 5b45b1aff7a0f246f5c2ff1bbc7bc4ac28c6fdd752633e3946e8f4976e34e40c
Sep 16 13:15:18.014602 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.014517    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"]
Sep 16 13:15:18.016677 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:15:18.016647    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0823a3d_37c3_408d_aed8_5b315b6f1381.slice/crio-20173bf7a89880c6d25549668a58e48e14473a401c46157b920633833edfc790 WatchSource:0}: Error finding container 20173bf7a89880c6d25549668a58e48e14473a401c46157b920633833edfc790: Status 404 returned error can't find the container with id 20173bf7a89880c6d25549668a58e48e14473a401c46157b920633833edfc790
Sep 16 13:15:18.199086 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.199048    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:15:18.233722 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.233684    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" event={"ID":"019e4224-cfaa-40fa-9eca-65aaee18a3df","Type":"ContainerStarted","Data":"a5479d429d2f0dd76731ff3871f9c78ce49517d691df90ab9a9f3b70f8e2e8b9"}
Sep 16 13:15:18.233916 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.233766    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:18.233916 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.233802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" event={"ID":"019e4224-cfaa-40fa-9eca-65aaee18a3df","Type":"ContainerStarted","Data":"9272600f6f52507d741149211a0235f003dd70261d67e48754fe05758f9cbd64"}
Sep 16 13:15:18.233916 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.233859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" event={"ID":"019e4224-cfaa-40fa-9eca-65aaee18a3df","Type":"ContainerStarted","Data":"5b45b1aff7a0f246f5c2ff1bbc7bc4ac28c6fdd752633e3946e8f4976e34e40c"}
Sep 16 13:15:18.235420 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.235394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" event={"ID":"a0823a3d-37c3-408d-aed8-5b315b6f1381","Type":"ContainerStarted","Data":"459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6"}
Sep 16 13:15:18.235539 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.235445    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" event={"ID":"a0823a3d-37c3-408d-aed8-5b315b6f1381","Type":"ContainerStarted","Data":"20173bf7a89880c6d25549668a58e48e14473a401c46157b920633833edfc790"}
Sep 16 13:15:18.235539 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.235492    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:18.256729 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.256613    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podStartSLOduration=2.256593106 podStartE2EDuration="2.256593106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:18.252637 +0000 UTC m=+1644.714265697" watchObservedRunningTime="2026-09-16 13:15:18.256593106 +0000 UTC m=+1644.718221802"
Sep 16 13:15:18.268635 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.268587    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podStartSLOduration=2.268571316 podStartE2EDuration="2.268571316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:18.268073722 +0000 UTC m=+1644.729702418" watchObservedRunningTime="2026-09-16 13:15:18.268571316 +0000 UTC m=+1644.730200012"
Sep 16 13:15:18.950304 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:18.950257    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 13:15:19.239593 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:19.239508    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:19.240757 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:19.240730    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 13:15:20.173437 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.173412    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:15:20.244529 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.244498    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b866478-1a7b-4b0d-8780-728008d47114" containerID="3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f" exitCode=0
Sep 16 13:15:20.244658 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.244540    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"
Sep 16 13:15:20.244658 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.244582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" event={"ID":"5b866478-1a7b-4b0d-8780-728008d47114","Type":"ContainerDied","Data":"3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f"}
Sep 16 13:15:20.244658 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.244642    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx" event={"ID":"5b866478-1a7b-4b0d-8780-728008d47114","Type":"ContainerDied","Data":"054d3a8bc66ba4a9f2ed45eda6856bdc503226709ad3f49537cdaa1a47bc1b06"}
Sep 16 13:15:20.244765 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.244668    2558 scope.go:117] "RemoveContainer" containerID="08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f"
Sep 16 13:15:20.245082 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.245059    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 13:15:20.253431 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.253416    2558 scope.go:117] "RemoveContainer" containerID="3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f"
Sep 16 13:15:20.260240 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.260225    2558 scope.go:117] "RemoveContainer" containerID="08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f"
Sep 16 13:15:20.260470 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:15:20.260452    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f\": container with ID starting with 08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f not found: ID does not exist" containerID="08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f"
Sep 16 13:15:20.260521 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.260479    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f"} err="failed to get container status \"08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f\": rpc error: code = NotFound desc = could not find container \"08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f\": container with ID starting with 08d53a3e8219a7de6800ea7377a4cd68d2695086e41458d593a4f5a8822ca30f not found: ID does not exist"
Sep 16 13:15:20.260521 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.260495    2558 scope.go:117] "RemoveContainer" containerID="3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f"
Sep 16 13:15:20.260755 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:15:20.260736    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f\": container with ID starting with 3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f not found: ID does not exist" containerID="3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f"
Sep 16 13:15:20.260829 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.260758    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f"} err="failed to get container status \"3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f\": rpc error: code = NotFound desc = could not find container \"3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f\": container with ID starting with 3949f65c56ef823658856c1b4cc9c89083ef0778aef00dc4a0a7ece46f02264f not found: ID does not exist"
Sep 16 13:15:20.317040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.316981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b866478-1a7b-4b0d-8780-728008d47114-kube-rbac-proxy-sar-config\") pod \"5b866478-1a7b-4b0d-8780-728008d47114\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") "
Sep 16 13:15:20.317040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.317038    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls\") pod \"5b866478-1a7b-4b0d-8780-728008d47114\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") "
Sep 16 13:15:20.317171 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.317067    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zv87\" (UniqueName: \"kubernetes.io/projected/5b866478-1a7b-4b0d-8780-728008d47114-kube-api-access-6zv87\") pod \"5b866478-1a7b-4b0d-8780-728008d47114\" (UID: \"5b866478-1a7b-4b0d-8780-728008d47114\") "
Sep 16 13:15:20.317327 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.317304    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b866478-1a7b-4b0d-8780-728008d47114-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b866478-1a7b-4b0d-8780-728008d47114" (UID: "5b866478-1a7b-4b0d-8780-728008d47114"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:15:20.319220 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.319192    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b866478-1a7b-4b0d-8780-728008d47114" (UID: "5b866478-1a7b-4b0d-8780-728008d47114"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:15:20.319220 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.319204    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b866478-1a7b-4b0d-8780-728008d47114-kube-api-access-6zv87" (OuterVolumeSpecName: "kube-api-access-6zv87") pod "5b866478-1a7b-4b0d-8780-728008d47114" (UID: "5b866478-1a7b-4b0d-8780-728008d47114"). InnerVolumeSpecName "kube-api-access-6zv87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:20.418088 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.418048    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b866478-1a7b-4b0d-8780-728008d47114-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:15:20.418088 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.418082    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b866478-1a7b-4b0d-8780-728008d47114-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:15:20.418088 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.418092    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zv87\" (UniqueName: \"kubernetes.io/projected/5b866478-1a7b-4b0d-8780-728008d47114-kube-api-access-6zv87\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:15:20.566027 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.565998    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"]
Sep 16 13:15:20.569489 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:20.569432    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa957-predictor-85ccdd9d88-88xfx"]
Sep 16 13:15:22.041481 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:22.041446    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b866478-1a7b-4b0d-8780-728008d47114" path="/var/lib/kubelet/pods/5b866478-1a7b-4b0d-8780-728008d47114/volumes"
Sep 16 13:15:23.198483 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:23.198444    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:15:24.245413 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:24.245385    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:15:25.250454 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:25.250369    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:15:25.254590 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:25.252543    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 13:15:28.198330 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:28.198290    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:15:28.200627 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:28.198401    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:33.198587 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:33.198549    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:15:35.253235 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:35.253194    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 13:15:38.198928 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:38.198881    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:15:43.199311 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:43.199268    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:15:45.253434 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:45.253385    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 13:15:46.661430 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:46.661399    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:46.730117 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:46.730081    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa02ad5-34fb-4042-9150-cc878e7b87ec-openshift-service-ca-bundle\") pod \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") "
Sep 16 13:15:46.730300 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:46.730153    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa02ad5-34fb-4042-9150-cc878e7b87ec-proxy-tls\") pod \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\" (UID: \"8fa02ad5-34fb-4042-9150-cc878e7b87ec\") "
Sep 16 13:15:46.730538 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:46.730496    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa02ad5-34fb-4042-9150-cc878e7b87ec-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8fa02ad5-34fb-4042-9150-cc878e7b87ec" (UID: "8fa02ad5-34fb-4042-9150-cc878e7b87ec"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:15:46.732232 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:46.732208    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa02ad5-34fb-4042-9150-cc878e7b87ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fa02ad5-34fb-4042-9150-cc878e7b87ec" (UID: "8fa02ad5-34fb-4042-9150-cc878e7b87ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:15:46.831334 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:46.831256    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa02ad5-34fb-4042-9150-cc878e7b87ec-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:15:46.831334 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:46.831286    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa02ad5-34fb-4042-9150-cc878e7b87ec-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:15:47.349728 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.349693    2558 generic.go:356] "Generic (PLEG): container finished" podID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerID="36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c" exitCode=0
Sep 16 13:15:47.350015 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.349757    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"
Sep 16 13:15:47.350015 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.349762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" event={"ID":"8fa02ad5-34fb-4042-9150-cc878e7b87ec","Type":"ContainerDied","Data":"36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c"}
Sep 16 13:15:47.350015 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.349802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk" event={"ID":"8fa02ad5-34fb-4042-9150-cc878e7b87ec","Type":"ContainerDied","Data":"408ee73ca113cb80648bcea4bdbdfb4e56f39ce5d0ed88dfb643cf00f669fdfb"}
Sep 16 13:15:47.350015 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.349842    2558 scope.go:117] "RemoveContainer" containerID="36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c"
Sep 16 13:15:47.358210 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.358195    2558 scope.go:117] "RemoveContainer" containerID="36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c"
Sep 16 13:15:47.358470 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:15:47.358451    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c\": container with ID starting with 36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c not found: ID does not exist" containerID="36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c"
Sep 16 13:15:47.358547 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.358476    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c"} err="failed to get container status \"36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c\": rpc error: code = NotFound desc = could not find container \"36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c\": container with ID starting with 36b2405516d5bce57d1c4388095295c8dc26c517c7a2a1b39e0d6d9ebb136e9c not found: ID does not exist"
Sep 16 13:15:47.370925 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.370902    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"]
Sep 16 13:15:47.375319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:47.375297    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-aa957-55d5b78657-vbgkk"]
Sep 16 13:15:48.041735 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:48.041694    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" path="/var/lib/kubelet/pods/8fa02ad5-34fb-4042-9150-cc878e7b87ec/volumes"
Sep 16 13:15:55.252804 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:15:55.252763    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 13:16:05.252976 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:05.252943    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:16:16.749548 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749515    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"]
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749834    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kube-rbac-proxy"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749854    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kube-rbac-proxy"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749869    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749875    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749891    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749897    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749942    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kserve-container"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749949    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b866478-1a7b-4b0d-8780-728008d47114" containerName="kube-rbac-proxy"
Sep 16 13:16:16.749993 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.749958    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa02ad5-34fb-4042-9150-cc878e7b87ec" containerName="sequence-graph-aa957"
Sep 16 13:16:16.752803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.752785    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:16.755201 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.755176    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4c3c9-kube-rbac-proxy-sar-config\""
Sep 16 13:16:16.755341 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.755295    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4c3c9-serving-cert\""
Sep 16 13:16:16.762803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.762778    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"]
Sep 16 13:16:16.883507 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.883465    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls\") pod \"sequence-graph-4c3c9-79ddd9c64c-2w8qj\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") " pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:16.883706 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.883550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06841234-5105-4354-ac6c-acbab8679818-openshift-service-ca-bundle\") pod \"sequence-graph-4c3c9-79ddd9c64c-2w8qj\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") " pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:16.984830 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.984781    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06841234-5105-4354-ac6c-acbab8679818-openshift-service-ca-bundle\") pod \"sequence-graph-4c3c9-79ddd9c64c-2w8qj\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") " pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:16.985046 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.984865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls\") pod \"sequence-graph-4c3c9-79ddd9c64c-2w8qj\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") " pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:16.985046 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:16:16.985024    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-4c3c9-serving-cert: secret "sequence-graph-4c3c9-serving-cert" not found
Sep 16 13:16:16.985176 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:16:16.985115    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls podName:06841234-5105-4354-ac6c-acbab8679818 nodeName:}" failed. No retries permitted until 2026-09-16 13:16:17.485090244 +0000 UTC m=+1703.946718919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls") pod "sequence-graph-4c3c9-79ddd9c64c-2w8qj" (UID: "06841234-5105-4354-ac6c-acbab8679818") : secret "sequence-graph-4c3c9-serving-cert" not found
Sep 16 13:16:16.985496 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:16.985475    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06841234-5105-4354-ac6c-acbab8679818-openshift-service-ca-bundle\") pod \"sequence-graph-4c3c9-79ddd9c64c-2w8qj\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") " pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:17.489925 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:17.489882    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls\") pod \"sequence-graph-4c3c9-79ddd9c64c-2w8qj\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") " pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:17.492419 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:17.492381    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls\") pod \"sequence-graph-4c3c9-79ddd9c64c-2w8qj\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") " pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:17.662987 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:17.662946    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:17.933779 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:17.933715    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"]
Sep 16 13:16:17.936650 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:16:17.936622    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06841234_5105_4354_ac6c_acbab8679818.slice/crio-d5a975a0cc53ad4bb2e4fd949a359d9c59782dc4299ed7378b64392ed66737dd WatchSource:0}: Error finding container d5a975a0cc53ad4bb2e4fd949a359d9c59782dc4299ed7378b64392ed66737dd: Status 404 returned error can't find the container with id d5a975a0cc53ad4bb2e4fd949a359d9c59782dc4299ed7378b64392ed66737dd
Sep 16 13:16:18.465546 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:18.465499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" event={"ID":"06841234-5105-4354-ac6c-acbab8679818","Type":"ContainerStarted","Data":"5fa50755b4cdf129921df94bcd67f9b52094731ffb1daef543928690887cb53c"}
Sep 16 13:16:18.465825 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:18.465558    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" event={"ID":"06841234-5105-4354-ac6c-acbab8679818","Type":"ContainerStarted","Data":"d5a975a0cc53ad4bb2e4fd949a359d9c59782dc4299ed7378b64392ed66737dd"}
Sep 16 13:16:18.465825 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:18.465638    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:16:18.482792 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:18.482739    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podStartSLOduration=2.482722981 podStartE2EDuration="2.482722981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:18.480307387 +0000 UTC m=+1704.941936082" watchObservedRunningTime="2026-09-16 13:16:18.482722981 +0000 UTC m=+1704.944351678"
Sep 16 13:16:24.475454 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:16:24.475422    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:23:31.486400 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.486362    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"]
Sep 16 13:23:31.489123 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.486580    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6" containerID="cri-o://459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6" gracePeriod=30
Sep 16 13:23:31.737833 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.737712    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"]
Sep 16 13:23:31.738033 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.737993    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" containerID="cri-o://e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6" gracePeriod=30
Sep 16 13:23:31.738110 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.738076    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kube-rbac-proxy" containerID="cri-o://d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1" gracePeriod=30
Sep 16 13:23:31.833968 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.833930    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"]
Sep 16 13:23:31.837261 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.837242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.839984 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.839949    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e1666-predictor-serving-cert\""
Sep 16 13:23:31.839984 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.839956    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e1666-kube-rbac-proxy-sar-config\""
Sep 16 13:23:31.849118 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.849090    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"]
Sep 16 13:23:31.882714 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.882674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nkv\" (UniqueName: \"kubernetes.io/projected/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-api-access-k9nkv\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.882946 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.882733    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.882946 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.882832    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-proxy-tls\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.984124 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.984080    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nkv\" (UniqueName: \"kubernetes.io/projected/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-api-access-k9nkv\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.984331 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.984140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.984331 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.984166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-proxy-tls\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.984962 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.984936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.986870 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.986846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-proxy-tls\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:31.992955 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:31.992893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nkv\" (UniqueName: \"kubernetes.io/projected/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-api-access-k9nkv\") pod \"error-404-isvc-e1666-predictor-67564d9658-2rhqz\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") " pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:32.147361 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:32.147324    2558 generic.go:356] "Generic (PLEG): container finished" podID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerID="d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1" exitCode=2
Sep 16 13:23:32.147519 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:32.147392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" event={"ID":"141a0ab6-e0a0-4d7f-b03a-c3484edce952","Type":"ContainerDied","Data":"d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1"}
Sep 16 13:23:32.147579 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:32.147555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:32.429069 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:32.428899    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"]
Sep 16 13:23:32.431977 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:23:32.431938    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fea6dae_aa00_4816_80c0_2f0ee3d9f7a1.slice/crio-eb7b3d215c052ad8072a17340ab8c60062a0106cece64601b688847ad96c5174 WatchSource:0}: Error finding container eb7b3d215c052ad8072a17340ab8c60062a0106cece64601b688847ad96c5174: Status 404 returned error can't find the container with id eb7b3d215c052ad8072a17340ab8c60062a0106cece64601b688847ad96c5174
Sep 16 13:23:33.154567 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:33.154525    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" event={"ID":"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1","Type":"ContainerStarted","Data":"5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1"}
Sep 16 13:23:33.155065 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:33.154605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" event={"ID":"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1","Type":"ContainerStarted","Data":"143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90"}
Sep 16 13:23:33.155065 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:33.154646    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:33.155065 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:33.154677    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" event={"ID":"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1","Type":"ContainerStarted","Data":"eb7b3d215c052ad8072a17340ab8c60062a0106cece64601b688847ad96c5174"}
Sep 16 13:23:33.179895 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:33.179839    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podStartSLOduration=2.1798050939999998 podStartE2EDuration="2.179805094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:33.178747333 +0000 UTC m=+2139.640376030" watchObservedRunningTime="2026-09-16 13:23:33.179805094 +0000 UTC m=+2139.641433790"
Sep 16 13:23:34.158837 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:34.158775    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:34.160381 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:34.160340    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 13:23:34.244132 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:34.244085    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:23:34.997681 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:34.997632    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 16 13:23:35.003105 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.003059    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 13:23:35.162536 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.162495    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 13:23:35.594284 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.594256    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:23:35.716241 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.716191    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a0ab6-e0a0-4d7f-b03a-c3484edce952-proxy-tls\") pod \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") "
Sep 16 13:23:35.716436 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.716252    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npz2s\" (UniqueName: \"kubernetes.io/projected/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-api-access-npz2s\") pod \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") "
Sep 16 13:23:35.716436 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.716306    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-rbac-proxy-sar-config\") pod \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\" (UID: \"141a0ab6-e0a0-4d7f-b03a-c3484edce952\") "
Sep 16 13:23:35.716705 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.716673    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "141a0ab6-e0a0-4d7f-b03a-c3484edce952" (UID: "141a0ab6-e0a0-4d7f-b03a-c3484edce952"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:23:35.718476 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.718448    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141a0ab6-e0a0-4d7f-b03a-c3484edce952-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "141a0ab6-e0a0-4d7f-b03a-c3484edce952" (UID: "141a0ab6-e0a0-4d7f-b03a-c3484edce952"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:35.718574 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.718506    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-api-access-npz2s" (OuterVolumeSpecName: "kube-api-access-npz2s") pod "141a0ab6-e0a0-4d7f-b03a-c3484edce952" (UID: "141a0ab6-e0a0-4d7f-b03a-c3484edce952"). InnerVolumeSpecName "kube-api-access-npz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:23:35.817940 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.817880    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a0ab6-e0a0-4d7f-b03a-c3484edce952-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:23:35.817940 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.817931    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz2s\" (UniqueName: \"kubernetes.io/projected/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-api-access-npz2s\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:23:35.817940 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:35.817942    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a0ab6-e0a0-4d7f-b03a-c3484edce952-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:23:36.167489 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.167451    2558 generic.go:356] "Generic (PLEG): container finished" podID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerID="e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6" exitCode=0
Sep 16 13:23:36.167963 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.167504    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"
Sep 16 13:23:36.167963 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.167540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" event={"ID":"141a0ab6-e0a0-4d7f-b03a-c3484edce952","Type":"ContainerDied","Data":"e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6"}
Sep 16 13:23:36.167963 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.167608    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4" event={"ID":"141a0ab6-e0a0-4d7f-b03a-c3484edce952","Type":"ContainerDied","Data":"efca864a8e92764e1b82572454219a1da0c67336f1785a3d467782876ec5f5e0"}
Sep 16 13:23:36.167963 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.167640    2558 scope.go:117] "RemoveContainer" containerID="d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1"
Sep 16 13:23:36.176292 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.176271    2558 scope.go:117] "RemoveContainer" containerID="e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6"
Sep 16 13:23:36.184404 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.184377    2558 scope.go:117] "RemoveContainer" containerID="d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1"
Sep 16 13:23:36.184891 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:23:36.184860    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1\": container with ID starting with d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1 not found: ID does not exist" containerID="d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1"
Sep 16 13:23:36.185014 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.184904    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1"} err="failed to get container status \"d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1\": rpc error: code = NotFound desc = could not find container \"d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1\": container with ID starting with d0ec5a2d7d39188c272af4bd736169a5475aeda1c44ced55073b2fb2804dbfd1 not found: ID does not exist"
Sep 16 13:23:36.185014 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.184930    2558 scope.go:117] "RemoveContainer" containerID="e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6"
Sep 16 13:23:36.185309 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:23:36.185288    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6\": container with ID starting with e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6 not found: ID does not exist" containerID="e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6"
Sep 16 13:23:36.185365 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.185315    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6"} err="failed to get container status \"e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6\": rpc error: code = NotFound desc = could not find container \"e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6\": container with ID starting with e68c4882a6ca1f4f83c8a484535c09c78dc88acd31293e28157c83af332694e6 not found: ID does not exist"
Sep 16 13:23:36.186585 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.186564    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"]
Sep 16 13:23:36.190594 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:36.190566    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30d6-predictor-b778cb6cb-xj7r4"]
Sep 16 13:23:38.042959 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:38.042920    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" path="/var/lib/kubelet/pods/141a0ab6-e0a0-4d7f-b03a-c3484edce952/volumes"
Sep 16 13:23:39.243741 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:39.243696    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:23:40.167257 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:40.167224    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:23:40.167877 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:40.167849    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 13:23:44.243410 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:44.243370    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:23:44.243859 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:44.243490    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:23:49.243999 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:49.243912    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:23:50.167980 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:50.167934    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 13:23:54.243015 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:54.242972    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:23:59.243630 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:23:59.243591    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:00.168096 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:00.168049    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 13:24:02.138912 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.138885    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:24:02.247239 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.247196    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0823a3d-37c3-408d-aed8-5b315b6f1381-openshift-service-ca-bundle\") pod \"a0823a3d-37c3-408d-aed8-5b315b6f1381\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") "
Sep 16 13:24:02.247561 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.247275    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls\") pod \"a0823a3d-37c3-408d-aed8-5b315b6f1381\" (UID: \"a0823a3d-37c3-408d-aed8-5b315b6f1381\") "
Sep 16 13:24:02.247681 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.247656    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0823a3d-37c3-408d-aed8-5b315b6f1381-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a0823a3d-37c3-408d-aed8-5b315b6f1381" (UID: "a0823a3d-37c3-408d-aed8-5b315b6f1381"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:24:02.249402 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.249375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0823a3d-37c3-408d-aed8-5b315b6f1381" (UID: "a0823a3d-37c3-408d-aed8-5b315b6f1381"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:24:02.273285 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.273252    2558 generic.go:356] "Generic (PLEG): container finished" podID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerID="459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6" exitCode=0
Sep 16 13:24:02.273452 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.273298    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"
Sep 16 13:24:02.273452 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.273334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" event={"ID":"a0823a3d-37c3-408d-aed8-5b315b6f1381","Type":"ContainerDied","Data":"459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6"}
Sep 16 13:24:02.273452 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.273392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p" event={"ID":"a0823a3d-37c3-408d-aed8-5b315b6f1381","Type":"ContainerDied","Data":"20173bf7a89880c6d25549668a58e48e14473a401c46157b920633833edfc790"}
Sep 16 13:24:02.273452 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.273420    2558 scope.go:117] "RemoveContainer" containerID="459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6"
Sep 16 13:24:02.281911 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.281887    2558 scope.go:117] "RemoveContainer" containerID="459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6"
Sep 16 13:24:02.282205 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:24:02.282183    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6\": container with ID starting with 459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6 not found: ID does not exist" containerID="459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6"
Sep 16 13:24:02.282257 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.282217    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6"} err="failed to get container status \"459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6\": rpc error: code = NotFound desc = could not find container \"459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6\": container with ID starting with 459d46337d91c1320e297eb9e5164b88c7cf2535724c1a953939b8a2719a7db6 not found: ID does not exist"
Sep 16 13:24:02.293725 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.293693    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"]
Sep 16 13:24:02.297450 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.297426    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d30d6-6486bfbb6f-42c7p"]
Sep 16 13:24:02.347942 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.347897    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0823a3d-37c3-408d-aed8-5b315b6f1381-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:02.347942 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:02.347934    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0823a3d-37c3-408d-aed8-5b315b6f1381-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:04.042786 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:04.042749    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" path="/var/lib/kubelet/pods/a0823a3d-37c3-408d-aed8-5b315b6f1381/volumes"
Sep 16 13:24:10.167872 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:10.167828    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 13:24:20.169165 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:20.169127    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:24:31.493247 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.493210    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"]
Sep 16 13:24:31.493691 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.493434    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9" containerID="cri-o://5fa50755b4cdf129921df94bcd67f9b52094731ffb1daef543928690887cb53c" gracePeriod=30
Sep 16 13:24:31.671741 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.671698    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"]
Sep 16 13:24:31.672069 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.672034    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" containerID="cri-o://9272600f6f52507d741149211a0235f003dd70261d67e48754fe05758f9cbd64" gracePeriod=30
Sep 16 13:24:31.672155 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.672111    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kube-rbac-proxy" containerID="cri-o://a5479d429d2f0dd76731ff3871f9c78ce49517d691df90ab9a9f3b70f8e2e8b9" gracePeriod=30
Sep 16 13:24:31.810621 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810586    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"]
Sep 16 13:24:31.810921 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810897    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6"
Sep 16 13:24:31.810921 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810913    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6"
Sep 16 13:24:31.810921 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810923    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kube-rbac-proxy"
Sep 16 13:24:31.810921 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810929    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kube-rbac-proxy"
Sep 16 13:24:31.811145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810948    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container"
Sep 16 13:24:31.811145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810954    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container"
Sep 16 13:24:31.811145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.810995    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0823a3d-37c3-408d-aed8-5b315b6f1381" containerName="ensemble-graph-d30d6"
Sep 16 13:24:31.811145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.811003    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kserve-container"
Sep 16 13:24:31.811145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.811009    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="141a0ab6-e0a0-4d7f-b03a-c3484edce952" containerName="kube-rbac-proxy"
Sep 16 13:24:31.813828 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.813781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:31.817797 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.817765    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-e1666-serving-cert\""
Sep 16 13:24:31.817797 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.817778    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-e1666-kube-rbac-proxy-sar-config\""
Sep 16 13:24:31.831292 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.831234    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"]
Sep 16 13:24:31.885938 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.885892    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"]
Sep 16 13:24:31.888930 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.888882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af79e7e-eef1-4568-b761-b9968b34ab54-proxy-tls\") pod \"splitter-graph-e1666-85477bf885-7h8mh\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") " pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:31.889122 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.888970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af79e7e-eef1-4568-b761-b9968b34ab54-openshift-service-ca-bundle\") pod \"splitter-graph-e1666-85477bf885-7h8mh\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") " pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:31.890069 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.890038    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:31.893061 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.893028    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a1820-predictor-serving-cert\""
Sep 16 13:24:31.893327 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.893302    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a1820-kube-rbac-proxy-sar-config\""
Sep 16 13:24:31.909161 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.909125    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"]
Sep 16 13:24:31.990118 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.990071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:31.990309 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.990126    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krtl\" (UniqueName: \"kubernetes.io/projected/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-api-access-8krtl\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:31.990309 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.990228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af79e7e-eef1-4568-b761-b9968b34ab54-proxy-tls\") pod \"splitter-graph-e1666-85477bf885-7h8mh\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") " pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:31.990309 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.990271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:31.990309 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.990299    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af79e7e-eef1-4568-b761-b9968b34ab54-openshift-service-ca-bundle\") pod \"splitter-graph-e1666-85477bf885-7h8mh\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") " pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:31.990985 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.990958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af79e7e-eef1-4568-b761-b9968b34ab54-openshift-service-ca-bundle\") pod \"splitter-graph-e1666-85477bf885-7h8mh\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") " pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:31.992753 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:31.992731    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af79e7e-eef1-4568-b761-b9968b34ab54-proxy-tls\") pod \"splitter-graph-e1666-85477bf885-7h8mh\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") " pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:32.091037 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.090934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:32.091037 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.090997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:32.091037 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.091021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krtl\" (UniqueName: \"kubernetes.io/projected/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-api-access-8krtl\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:32.091320 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:24:32.091112    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a1820-predictor-serving-cert: secret "error-404-isvc-a1820-predictor-serving-cert" not found
Sep 16 13:24:32.091320 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:24:32.091192    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls podName:a51c368d-7fb5-4c59-905f-784565b4d1f9 nodeName:}" failed. No retries permitted until 2026-09-16 13:24:32.591174468 +0000 UTC m=+2199.052803155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls") pod "error-404-isvc-a1820-predictor-79bf8db794-hl7vl" (UID: "a51c368d-7fb5-4c59-905f-784565b4d1f9") : secret "error-404-isvc-a1820-predictor-serving-cert" not found
Sep 16 13:24:32.091764 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.091743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:32.101171 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.101142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krtl\" (UniqueName: \"kubernetes.io/projected/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-api-access-8krtl\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:32.124981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.124939    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:32.395119 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.395081    2558 generic.go:356] "Generic (PLEG): container finished" podID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerID="a5479d429d2f0dd76731ff3871f9c78ce49517d691df90ab9a9f3b70f8e2e8b9" exitCode=2
Sep 16 13:24:32.395315 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.395146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" event={"ID":"019e4224-cfaa-40fa-9eca-65aaee18a3df","Type":"ContainerDied","Data":"a5479d429d2f0dd76731ff3871f9c78ce49517d691df90ab9a9f3b70f8e2e8b9"}
Sep 16 13:24:32.404020 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.403993    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"]
Sep 16 13:24:32.406619 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:24:32.406590    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af79e7e_eef1_4568_b761_b9968b34ab54.slice/crio-d61a343165a12d8391b4d480501643221add3c70c7f193a476a5b742befd230a WatchSource:0}: Error finding container d61a343165a12d8391b4d480501643221add3c70c7f193a476a5b742befd230a: Status 404 returned error can't find the container with id d61a343165a12d8391b4d480501643221add3c70c7f193a476a5b742befd230a
Sep 16 13:24:32.594644 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.594609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:32.597119 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.597096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls\") pod \"error-404-isvc-a1820-predictor-79bf8db794-hl7vl\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") " pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:32.802032 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:32.801865    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:33.089489 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.089397    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"]
Sep 16 13:24:33.092178 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:24:33.092151    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51c368d_7fb5_4c59_905f_784565b4d1f9.slice/crio-39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04 WatchSource:0}: Error finding container 39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04: Status 404 returned error can't find the container with id 39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04
Sep 16 13:24:33.401328 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.401288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" event={"ID":"a51c368d-7fb5-4c59-905f-784565b4d1f9","Type":"ContainerStarted","Data":"f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"}
Sep 16 13:24:33.401644 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.401469    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:33.401644 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.401584    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:33.401768 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.401648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" event={"ID":"a51c368d-7fb5-4c59-905f-784565b4d1f9","Type":"ContainerStarted","Data":"fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"}
Sep 16 13:24:33.401768 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.401700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" event={"ID":"a51c368d-7fb5-4c59-905f-784565b4d1f9","Type":"ContainerStarted","Data":"39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04"}
Sep 16 13:24:33.403157 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.403121    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 13:24:33.403322 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.403180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" event={"ID":"4af79e7e-eef1-4568-b761-b9968b34ab54","Type":"ContainerStarted","Data":"5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696"}
Sep 16 13:24:33.403322 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.403228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" event={"ID":"4af79e7e-eef1-4568-b761-b9968b34ab54","Type":"ContainerStarted","Data":"d61a343165a12d8391b4d480501643221add3c70c7f193a476a5b742befd230a"}
Sep 16 13:24:33.403322 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.403258    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:33.431358 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.431299    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podStartSLOduration=2.431281539 podStartE2EDuration="2.431281539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:33.430097918 +0000 UTC m=+2199.891726616" watchObservedRunningTime="2026-09-16 13:24:33.431281539 +0000 UTC m=+2199.892910235"
Sep 16 13:24:33.463201 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:33.463147    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podStartSLOduration=2.463128565 podStartE2EDuration="2.463128565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:33.462770471 +0000 UTC m=+2199.924399167" watchObservedRunningTime="2026-09-16 13:24:33.463128565 +0000 UTC m=+2199.924757261"
Sep 16 13:24:34.407260 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:34.407222    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 13:24:34.473618 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:34.473573    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:35.245536 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.245483    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 16 13:24:35.252532 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.252498    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 13:24:35.413831 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.413775    2558 generic.go:356] "Generic (PLEG): container finished" podID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerID="9272600f6f52507d741149211a0235f003dd70261d67e48754fe05758f9cbd64" exitCode=0
Sep 16 13:24:35.414230 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.413859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" event={"ID":"019e4224-cfaa-40fa-9eca-65aaee18a3df","Type":"ContainerDied","Data":"9272600f6f52507d741149211a0235f003dd70261d67e48754fe05758f9cbd64"}
Sep 16 13:24:35.532562 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.532528    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:24:35.622332 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.622287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls\") pod \"019e4224-cfaa-40fa-9eca-65aaee18a3df\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") "
Sep 16 13:24:35.622514 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.622354    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjtqk\" (UniqueName: \"kubernetes.io/projected/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-api-access-zjtqk\") pod \"019e4224-cfaa-40fa-9eca-65aaee18a3df\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") "
Sep 16 13:24:35.622514 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.622390    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-rbac-proxy-sar-config\") pod \"019e4224-cfaa-40fa-9eca-65aaee18a3df\" (UID: \"019e4224-cfaa-40fa-9eca-65aaee18a3df\") "
Sep 16 13:24:35.622795 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.622769    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "019e4224-cfaa-40fa-9eca-65aaee18a3df" (UID: "019e4224-cfaa-40fa-9eca-65aaee18a3df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:24:35.624510 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.624480    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-api-access-zjtqk" (OuterVolumeSpecName: "kube-api-access-zjtqk") pod "019e4224-cfaa-40fa-9eca-65aaee18a3df" (UID: "019e4224-cfaa-40fa-9eca-65aaee18a3df"). InnerVolumeSpecName "kube-api-access-zjtqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:35.624579 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.624498    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "019e4224-cfaa-40fa-9eca-65aaee18a3df" (UID: "019e4224-cfaa-40fa-9eca-65aaee18a3df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:24:35.723104 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.723047    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:35.723104 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.723098    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/019e4224-cfaa-40fa-9eca-65aaee18a3df-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:35.723104 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:35.723115    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtqk\" (UniqueName: \"kubernetes.io/projected/019e4224-cfaa-40fa-9eca-65aaee18a3df-kube-api-access-zjtqk\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:36.419165 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:36.419135    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"
Sep 16 13:24:36.419614 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:36.419134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5" event={"ID":"019e4224-cfaa-40fa-9eca-65aaee18a3df","Type":"ContainerDied","Data":"5b45b1aff7a0f246f5c2ff1bbc7bc4ac28c6fdd752633e3946e8f4976e34e40c"}
Sep 16 13:24:36.419614 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:36.419274    2558 scope.go:117] "RemoveContainer" containerID="a5479d429d2f0dd76731ff3871f9c78ce49517d691df90ab9a9f3b70f8e2e8b9"
Sep 16 13:24:36.427535 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:36.427514    2558 scope.go:117] "RemoveContainer" containerID="9272600f6f52507d741149211a0235f003dd70261d67e48754fe05758f9cbd64"
Sep 16 13:24:36.439358 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:36.439329    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"]
Sep 16 13:24:36.441335 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:36.441310    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c3c9-predictor-5cdbd6cff4-ms4k5"]
Sep 16 13:24:38.042238 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:38.042199    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" path="/var/lib/kubelet/pods/019e4224-cfaa-40fa-9eca-65aaee18a3df/volumes"
Sep 16 13:24:39.412381 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:39.412349    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:24:39.412887 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:39.412865    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:39.412992 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:39.412940    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 13:24:39.475144 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:39.475103    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:41.895039 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:41.895002    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"]
Sep 16 13:24:41.895475 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:41.895247    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666" containerID="cri-o://5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696" gracePeriod=30
Sep 16 13:24:42.047359 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.047325    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"]
Sep 16 13:24:42.047621 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.047565    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container" containerID="cri-o://143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90" gracePeriod=30
Sep 16 13:24:42.047621 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.047582    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kube-rbac-proxy" containerID="cri-o://5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1" gracePeriod=30
Sep 16 13:24:42.222554 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.222517    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"]
Sep 16 13:24:42.222871 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.222859    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container"
Sep 16 13:24:42.222922 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.222873    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container"
Sep 16 13:24:42.222922 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.222888    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kube-rbac-proxy"
Sep 16 13:24:42.222922 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.222894    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kube-rbac-proxy"
Sep 16 13:24:42.223018 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.222952    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kserve-container"
Sep 16 13:24:42.223018 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.222959    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="019e4224-cfaa-40fa-9eca-65aaee18a3df" containerName="kube-rbac-proxy"
Sep 16 13:24:42.225992 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.225972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.228711 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.228686    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-11631-predictor-serving-cert\""
Sep 16 13:24:42.229088 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.229068    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-11631-kube-rbac-proxy-sar-config\""
Sep 16 13:24:42.236506 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.236474    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"]
Sep 16 13:24:42.384036 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.383991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.384225 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.384052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.384225 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.384118    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zmj\" (UniqueName: \"kubernetes.io/projected/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-api-access-j6zmj\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.447585 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.447479    2558 generic.go:356] "Generic (PLEG): container finished" podID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerID="5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1" exitCode=2
Sep 16 13:24:42.447585 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.447560    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" event={"ID":"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1","Type":"ContainerDied","Data":"5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1"}
Sep 16 13:24:42.485488 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.485443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.485488 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.485493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.485728 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.485531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zmj\" (UniqueName: \"kubernetes.io/projected/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-api-access-j6zmj\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.485728 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:24:42.485601    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-11631-predictor-serving-cert: secret "error-404-isvc-11631-predictor-serving-cert" not found
Sep 16 13:24:42.485728 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:24:42.485681    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls podName:d06f5a66-69c5-4c5d-9419-d76fcf88b6e7 nodeName:}" failed. No retries permitted until 2026-09-16 13:24:42.985662479 +0000 UTC m=+2209.447291152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls") pod "error-404-isvc-11631-predictor-58dc68f8c4-spbkr" (UID: "d06f5a66-69c5-4c5d-9419-d76fcf88b6e7") : secret "error-404-isvc-11631-predictor-serving-cert" not found
Sep 16 13:24:42.486313 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.486294    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.495192 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.495160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zmj\" (UniqueName: \"kubernetes.io/projected/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-api-access-j6zmj\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.989189 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.989137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:42.991627 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:42.991603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls\") pod \"error-404-isvc-11631-predictor-58dc68f8c4-spbkr\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") " pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:43.136678 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:43.136627    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:43.420118 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:43.420082    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"]
Sep 16 13:24:43.423344 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:24:43.423307    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06f5a66_69c5_4c5d_9419_d76fcf88b6e7.slice/crio-54875481fd325ae108ba3105460c778a23d0bb50baf4dbaa807a3b6eb023d30e WatchSource:0}: Error finding container 54875481fd325ae108ba3105460c778a23d0bb50baf4dbaa807a3b6eb023d30e: Status 404 returned error can't find the container with id 54875481fd325ae108ba3105460c778a23d0bb50baf4dbaa807a3b6eb023d30e
Sep 16 13:24:43.453390 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:43.453352    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" event={"ID":"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7","Type":"ContainerStarted","Data":"54875481fd325ae108ba3105460c778a23d0bb50baf4dbaa807a3b6eb023d30e"}
Sep 16 13:24:44.411554 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:44.411508    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:44.461279 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:44.461233    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" event={"ID":"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7","Type":"ContainerStarted","Data":"2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff"}
Sep 16 13:24:44.461491 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:44.461311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" event={"ID":"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7","Type":"ContainerStarted","Data":"c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8"}
Sep 16 13:24:44.461491 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:44.461346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:44.473739 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:44.473696    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:44.473969 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:44.473856    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:24:44.483072 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:44.483016    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podStartSLOduration=2.482997462 podStartE2EDuration="2.482997462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:44.48285652 +0000 UTC m=+2210.944485211" watchObservedRunningTime="2026-09-16 13:24:44.482997462 +0000 UTC m=+2210.944626158"
Sep 16 13:24:45.163727 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:45.163685    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 16 13:24:45.465047 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:45.464959    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:45.466433 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:45.466401    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 13:24:45.922290 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:45.922258    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:24:46.015009 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.014967    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9nkv\" (UniqueName: \"kubernetes.io/projected/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-api-access-k9nkv\") pod \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") "
Sep 16 13:24:46.015198 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.015103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-proxy-tls\") pod \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") "
Sep 16 13:24:46.015198 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.015146    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-rbac-proxy-sar-config\") pod \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\" (UID: \"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1\") "
Sep 16 13:24:46.015592 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.015530    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" (UID: "6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:24:46.017145 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.017116    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" (UID: "6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:24:46.017244 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.017184    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-api-access-k9nkv" (OuterVolumeSpecName: "kube-api-access-k9nkv") pod "6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" (UID: "6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1"). InnerVolumeSpecName "kube-api-access-k9nkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:46.115981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.115938    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.115981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.115974    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.115981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.115984    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nkv\" (UniqueName: \"kubernetes.io/projected/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1-kube-api-access-k9nkv\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.470748 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.470706    2558 generic.go:356] "Generic (PLEG): container finished" podID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerID="143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90" exitCode=0
Sep 16 13:24:46.470748 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.470745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" event={"ID":"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1","Type":"ContainerDied","Data":"143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90"}
Sep 16 13:24:46.471302 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.470759    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"
Sep 16 13:24:46.471302 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.470829    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz" event={"ID":"6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1","Type":"ContainerDied","Data":"eb7b3d215c052ad8072a17340ab8c60062a0106cece64601b688847ad96c5174"}
Sep 16 13:24:46.471302 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.470857    2558 scope.go:117] "RemoveContainer" containerID="5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1"
Sep 16 13:24:46.471633 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.471604    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 13:24:46.480983 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.480962    2558 scope.go:117] "RemoveContainer" containerID="143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90"
Sep 16 13:24:46.488880 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.488825    2558 scope.go:117] "RemoveContainer" containerID="5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1"
Sep 16 13:24:46.489293 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:24:46.489263    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1\": container with ID starting with 5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1 not found: ID does not exist" containerID="5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1"
Sep 16 13:24:46.489425 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.489304    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1"} err="failed to get container status \"5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1\": rpc error: code = NotFound desc = could not find container \"5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1\": container with ID starting with 5eefcfa60748a04bb646ea99aa5bdd73af769198a638e74bbe026feaa899d7f1 not found: ID does not exist"
Sep 16 13:24:46.489425 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.489328    2558 scope.go:117] "RemoveContainer" containerID="143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90"
Sep 16 13:24:46.489698 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:24:46.489671    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90\": container with ID starting with 143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90 not found: ID does not exist" containerID="143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90"
Sep 16 13:24:46.489864 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.489706    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90"} err="failed to get container status \"143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90\": rpc error: code = NotFound desc = could not find container \"143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90\": container with ID starting with 143aa5b13b182187121e97f3e3a0b06080e7e75f2e7b3206514abaf6af055f90 not found: ID does not exist"
Sep 16 13:24:46.491319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.491298    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"]
Sep 16 13:24:46.499130 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:46.499104    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e1666-predictor-67564d9658-2rhqz"]
Sep 16 13:24:48.042347 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:48.042310    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" path="/var/lib/kubelet/pods/6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1/volumes"
Sep 16 13:24:49.411796 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:49.411753    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:49.413195 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:49.413169    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 13:24:49.474449 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:49.474406    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:51.476474 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:51.476439    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:24:51.476979 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:51.476948    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 13:24:54.411490 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:54.411445    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:54.411983 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:54.411562    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:24:54.473573 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:54.473479    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:59.411171 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:59.411132    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:59.413382 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:59.413357    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 13:24:59.473940 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:24:59.473902    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:25:01.477780 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.477728    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 13:25:01.541138 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.541098    2558 generic.go:356] "Generic (PLEG): container finished" podID="06841234-5105-4354-ac6c-acbab8679818" containerID="5fa50755b4cdf129921df94bcd67f9b52094731ffb1daef543928690887cb53c" exitCode=0
Sep 16 13:25:01.541312 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.541162    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" event={"ID":"06841234-5105-4354-ac6c-acbab8679818","Type":"ContainerDied","Data":"5fa50755b4cdf129921df94bcd67f9b52094731ffb1daef543928690887cb53c"}
Sep 16 13:25:01.647478 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.647447    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:25:01.749153 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.749057    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls\") pod \"06841234-5105-4354-ac6c-acbab8679818\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") "
Sep 16 13:25:01.749153 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.749122    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06841234-5105-4354-ac6c-acbab8679818-openshift-service-ca-bundle\") pod \"06841234-5105-4354-ac6c-acbab8679818\" (UID: \"06841234-5105-4354-ac6c-acbab8679818\") "
Sep 16 13:25:01.749474 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.749448    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06841234-5105-4354-ac6c-acbab8679818-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "06841234-5105-4354-ac6c-acbab8679818" (UID: "06841234-5105-4354-ac6c-acbab8679818"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:25:01.751367 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.751321    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06841234-5105-4354-ac6c-acbab8679818" (UID: "06841234-5105-4354-ac6c-acbab8679818"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:25:01.849803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.849751    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06841234-5105-4354-ac6c-acbab8679818-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:25:01.849803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:01.849797    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06841234-5105-4354-ac6c-acbab8679818-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:25:02.546651 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:02.546619    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"
Sep 16 13:25:02.547113 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:02.546619    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj" event={"ID":"06841234-5105-4354-ac6c-acbab8679818","Type":"ContainerDied","Data":"d5a975a0cc53ad4bb2e4fd949a359d9c59782dc4299ed7378b64392ed66737dd"}
Sep 16 13:25:02.547113 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:02.546757    2558 scope.go:117] "RemoveContainer" containerID="5fa50755b4cdf129921df94bcd67f9b52094731ffb1daef543928690887cb53c"
Sep 16 13:25:02.568171 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:02.568141    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"]
Sep 16 13:25:02.575324 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:02.575297    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4c3c9-79ddd9c64c-2w8qj"]
Sep 16 13:25:04.042177 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:04.042134    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06841234-5105-4354-ac6c-acbab8679818" path="/var/lib/kubelet/pods/06841234-5105-4354-ac6c-acbab8679818/volumes"
Sep 16 13:25:04.411443 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:04.411399    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:25:09.411442 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:09.411393    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:25:09.412975 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:09.412947    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 13:25:11.477132 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:11.477085    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 13:25:12.046761 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.046731    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:25:12.131021 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.130982    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af79e7e-eef1-4568-b761-b9968b34ab54-openshift-service-ca-bundle\") pod \"4af79e7e-eef1-4568-b761-b9968b34ab54\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") "
Sep 16 13:25:12.131209 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.131048    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af79e7e-eef1-4568-b761-b9968b34ab54-proxy-tls\") pod \"4af79e7e-eef1-4568-b761-b9968b34ab54\" (UID: \"4af79e7e-eef1-4568-b761-b9968b34ab54\") "
Sep 16 13:25:12.131326 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.131302    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af79e7e-eef1-4568-b761-b9968b34ab54-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4af79e7e-eef1-4568-b761-b9968b34ab54" (UID: "4af79e7e-eef1-4568-b761-b9968b34ab54"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:25:12.133137 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.133109    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af79e7e-eef1-4568-b761-b9968b34ab54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4af79e7e-eef1-4568-b761-b9968b34ab54" (UID: "4af79e7e-eef1-4568-b761-b9968b34ab54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:25:12.232371 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.232249    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af79e7e-eef1-4568-b761-b9968b34ab54-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:25:12.232371 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.232285    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af79e7e-eef1-4568-b761-b9968b34ab54-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:25:12.588003 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.587912    2558 generic.go:356] "Generic (PLEG): container finished" podID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerID="5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696" exitCode=0
Sep 16 13:25:12.588003 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.587979    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"
Sep 16 13:25:12.588003 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.587987    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" event={"ID":"4af79e7e-eef1-4568-b761-b9968b34ab54","Type":"ContainerDied","Data":"5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696"}
Sep 16 13:25:12.588535 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.588055    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh" event={"ID":"4af79e7e-eef1-4568-b761-b9968b34ab54","Type":"ContainerDied","Data":"d61a343165a12d8391b4d480501643221add3c70c7f193a476a5b742befd230a"}
Sep 16 13:25:12.588535 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.588077    2558 scope.go:117] "RemoveContainer" containerID="5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696"
Sep 16 13:25:12.596607 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.596580    2558 scope.go:117] "RemoveContainer" containerID="5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696"
Sep 16 13:25:12.596995 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:25:12.596973    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696\": container with ID starting with 5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696 not found: ID does not exist" containerID="5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696"
Sep 16 13:25:12.597062 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.597006    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696"} err="failed to get container status \"5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696\": rpc error: code = NotFound desc = could not find container \"5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696\": container with ID starting with 5f11e16ffd7a9b3c4fb3d4cfc896688b4757bd44204c981e88f62f5272833696 not found: ID does not exist"
Sep 16 13:25:12.610969 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.610928    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"]
Sep 16 13:25:12.616025 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:12.615991    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e1666-85477bf885-7h8mh"]
Sep 16 13:25:14.041668 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:14.041628    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" path="/var/lib/kubelet/pods/4af79e7e-eef1-4568-b761-b9968b34ab54/volumes"
Sep 16 13:25:19.413230 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:19.413142    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 13:25:21.477457 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:21.477411    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 13:25:29.413949 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:29.413915    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:25:31.478580 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.478547    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:25:31.753501 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753406    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"]
Sep 16 13:25:31.753726 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753713    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container"
Sep 16 13:25:31.753769 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753728    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container"
Sep 16 13:25:31.753769 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753739    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kube-rbac-proxy"
Sep 16 13:25:31.753769 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753746    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kube-rbac-proxy"
Sep 16 13:25:31.753769 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753756    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666"
Sep 16 13:25:31.753769 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753762    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666"
Sep 16 13:25:31.753769 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753769    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9"
Sep 16 13:25:31.753990 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753774    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9"
Sep 16 13:25:31.753990 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753881    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af79e7e-eef1-4568-b761-b9968b34ab54" containerName="splitter-graph-e1666"
Sep 16 13:25:31.753990 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753891    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="06841234-5105-4354-ac6c-acbab8679818" containerName="sequence-graph-4c3c9"
Sep 16 13:25:31.753990 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753897    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kube-rbac-proxy"
Sep 16 13:25:31.753990 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.753904    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fea6dae-aa00-4816-80c0-2f0ee3d9f7a1" containerName="kserve-container"
Sep 16 13:25:31.756658 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.756639    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:31.759315 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.759291    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a1820-kube-rbac-proxy-sar-config\""
Sep 16 13:25:31.760036 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.760018    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a1820-serving-cert\""
Sep 16 13:25:31.773999 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.773964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"]
Sep 16 13:25:31.792164 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.792131    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls\") pod \"switch-graph-a1820-864897f7c9-6pkdx\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") " pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:31.792363 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.792180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40e56f7c-0ced-489f-8b30-182c21fd80a2-openshift-service-ca-bundle\") pod \"switch-graph-a1820-864897f7c9-6pkdx\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") " pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:31.893555 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.893514    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls\") pod \"switch-graph-a1820-864897f7c9-6pkdx\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") " pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:31.893737 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.893574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40e56f7c-0ced-489f-8b30-182c21fd80a2-openshift-service-ca-bundle\") pod \"switch-graph-a1820-864897f7c9-6pkdx\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") " pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:31.893737 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:25:31.893674    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-a1820-serving-cert: secret "switch-graph-a1820-serving-cert" not found
Sep 16 13:25:31.893843 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:25:31.893742    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls podName:40e56f7c-0ced-489f-8b30-182c21fd80a2 nodeName:}" failed. No retries permitted until 2026-09-16 13:25:32.39372621 +0000 UTC m=+2258.855354887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls") pod "switch-graph-a1820-864897f7c9-6pkdx" (UID: "40e56f7c-0ced-489f-8b30-182c21fd80a2") : secret "switch-graph-a1820-serving-cert" not found
Sep 16 13:25:31.894223 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:31.894205    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40e56f7c-0ced-489f-8b30-182c21fd80a2-openshift-service-ca-bundle\") pod \"switch-graph-a1820-864897f7c9-6pkdx\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") " pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:32.397279 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:32.397233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls\") pod \"switch-graph-a1820-864897f7c9-6pkdx\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") " pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:32.399767 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:32.399728    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls\") pod \"switch-graph-a1820-864897f7c9-6pkdx\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") " pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:32.667394 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:32.667297    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:32.956028 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:32.955783    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"]
Sep 16 13:25:32.958660 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:25:32.958631    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e56f7c_0ced_489f_8b30_182c21fd80a2.slice/crio-853c43e458129eb87aec9234006164e418025d7a59241e69d2f3f411208e5b3c WatchSource:0}: Error finding container 853c43e458129eb87aec9234006164e418025d7a59241e69d2f3f411208e5b3c: Status 404 returned error can't find the container with id 853c43e458129eb87aec9234006164e418025d7a59241e69d2f3f411208e5b3c
Sep 16 13:25:33.670761 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:33.670723    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" event={"ID":"40e56f7c-0ced-489f-8b30-182c21fd80a2","Type":"ContainerStarted","Data":"44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863"}
Sep 16 13:25:33.671221 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:33.670783    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" event={"ID":"40e56f7c-0ced-489f-8b30-182c21fd80a2","Type":"ContainerStarted","Data":"853c43e458129eb87aec9234006164e418025d7a59241e69d2f3f411208e5b3c"}
Sep 16 13:25:33.671221 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:33.670845    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:33.696213 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:33.696157    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podStartSLOduration=2.696141454 podStartE2EDuration="2.696141454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:25:33.695968088 +0000 UTC m=+2260.157596784" watchObservedRunningTime="2026-09-16 13:25:33.696141454 +0000 UTC m=+2260.157770193"
Sep 16 13:25:39.681305 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:39.681273    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:25:42.157139 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.157084    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"]
Sep 16 13:25:42.161664 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.161634    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:42.164381 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.164345    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-11631-kube-rbac-proxy-sar-config\""
Sep 16 13:25:42.164564 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.164468    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-11631-serving-cert\""
Sep 16 13:25:42.172469 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.172440    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"]
Sep 16 13:25:42.285076 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.285033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/265defd5-6763-430d-8133-9261ea2bcfd8-openshift-service-ca-bundle\") pod \"splitter-graph-11631-6b777c6c96-4j7bp\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") " pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:42.285265 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.285094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls\") pod \"splitter-graph-11631-6b777c6c96-4j7bp\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") " pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:42.386441 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.386376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/265defd5-6763-430d-8133-9261ea2bcfd8-openshift-service-ca-bundle\") pod \"splitter-graph-11631-6b777c6c96-4j7bp\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") " pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:42.386666 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.386532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls\") pod \"splitter-graph-11631-6b777c6c96-4j7bp\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") " pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:42.386725 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:25:42.386685    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-11631-serving-cert: secret "splitter-graph-11631-serving-cert" not found
Sep 16 13:25:42.386784 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:25:42.386771    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls podName:265defd5-6763-430d-8133-9261ea2bcfd8 nodeName:}" failed. No retries permitted until 2026-09-16 13:25:42.886745898 +0000 UTC m=+2269.348374587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls") pod "splitter-graph-11631-6b777c6c96-4j7bp" (UID: "265defd5-6763-430d-8133-9261ea2bcfd8") : secret "splitter-graph-11631-serving-cert" not found
Sep 16 13:25:42.387233 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.387213    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/265defd5-6763-430d-8133-9261ea2bcfd8-openshift-service-ca-bundle\") pod \"splitter-graph-11631-6b777c6c96-4j7bp\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") " pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:42.891899 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.891846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls\") pod \"splitter-graph-11631-6b777c6c96-4j7bp\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") " pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:42.894408 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:42.894374    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls\") pod \"splitter-graph-11631-6b777c6c96-4j7bp\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") " pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:43.073645 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:43.073584    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:43.357261 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:43.357169    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"]
Sep 16 13:25:43.362699 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:25:43.362670    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265defd5_6763_430d_8133_9261ea2bcfd8.slice/crio-98217551c6dc930d7653a46c76a12d5b853fa9a2ff3d4cbfc3694420e3852aeb WatchSource:0}: Error finding container 98217551c6dc930d7653a46c76a12d5b853fa9a2ff3d4cbfc3694420e3852aeb: Status 404 returned error can't find the container with id 98217551c6dc930d7653a46c76a12d5b853fa9a2ff3d4cbfc3694420e3852aeb
Sep 16 13:25:43.711832 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:43.711772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" event={"ID":"265defd5-6763-430d-8133-9261ea2bcfd8","Type":"ContainerStarted","Data":"d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a"}
Sep 16 13:25:43.712022 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:43.711869    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:25:43.712022 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:43.711894    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" event={"ID":"265defd5-6763-430d-8133-9261ea2bcfd8","Type":"ContainerStarted","Data":"98217551c6dc930d7653a46c76a12d5b853fa9a2ff3d4cbfc3694420e3852aeb"}
Sep 16 13:25:43.747099 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:43.747039    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podStartSLOduration=1.747018865 podStartE2EDuration="1.747018865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:25:43.745789668 +0000 UTC m=+2270.207418365" watchObservedRunningTime="2026-09-16 13:25:43.747018865 +0000 UTC m=+2270.208647560"
Sep 16 13:25:49.722367 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:25:49.722335    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:33:56.714970 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:56.714934    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"]
Sep 16 13:33:56.717690 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:56.715212    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631" containerID="cri-o://d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a" gracePeriod=30
Sep 16 13:33:56.866404 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:56.866372    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"]
Sep 16 13:33:56.866650 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:56.866610    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container" containerID="cri-o://c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8" gracePeriod=30
Sep 16 13:33:56.866770 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:56.866711    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kube-rbac-proxy" containerID="cri-o://2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff" gracePeriod=30
Sep 16 13:33:57.586784 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:57.586749    2558 generic.go:356] "Generic (PLEG): container finished" podID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerID="2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff" exitCode=2
Sep 16 13:33:57.586977 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:57.586832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" event={"ID":"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7","Type":"ContainerDied","Data":"2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff"}
Sep 16 13:33:59.719873 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.719832    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:33:59.901786 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.901761    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:33:59.913324 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.913299    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-rbac-proxy-sar-config\") pod \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") "
Sep 16 13:33:59.913458 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.913332    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls\") pod \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") "
Sep 16 13:33:59.913458 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.913365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6zmj\" (UniqueName: \"kubernetes.io/projected/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-api-access-j6zmj\") pod \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\" (UID: \"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7\") "
Sep 16 13:33:59.913687 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.913662    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" (UID: "d06f5a66-69c5-4c5d-9419-d76fcf88b6e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:33:59.915369 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.915341    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" (UID: "d06f5a66-69c5-4c5d-9419-d76fcf88b6e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:33:59.915468 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:33:59.915391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-api-access-j6zmj" (OuterVolumeSpecName: "kube-api-access-j6zmj") pod "d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" (UID: "d06f5a66-69c5-4c5d-9419-d76fcf88b6e7"). InnerVolumeSpecName "kube-api-access-j6zmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:34:00.013929 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.013845    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:34:00.013929 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.013872    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:34:00.013929 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.013883    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zmj\" (UniqueName: \"kubernetes.io/projected/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7-kube-api-access-j6zmj\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:34:00.599576 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.599540    2558 generic.go:356] "Generic (PLEG): container finished" podID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerID="c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8" exitCode=0
Sep 16 13:34:00.599747 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.599588    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"
Sep 16 13:34:00.599747 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.599621    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" event={"ID":"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7","Type":"ContainerDied","Data":"c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8"}
Sep 16 13:34:00.599747 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.599670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr" event={"ID":"d06f5a66-69c5-4c5d-9419-d76fcf88b6e7","Type":"ContainerDied","Data":"54875481fd325ae108ba3105460c778a23d0bb50baf4dbaa807a3b6eb023d30e"}
Sep 16 13:34:00.599747 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.599698    2558 scope.go:117] "RemoveContainer" containerID="2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff"
Sep 16 13:34:00.607465 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.607446    2558 scope.go:117] "RemoveContainer" containerID="c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8"
Sep 16 13:34:00.614739 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.614713    2558 scope.go:117] "RemoveContainer" containerID="2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff"
Sep 16 13:34:00.615137 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:34:00.615111    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff\": container with ID starting with 2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff not found: ID does not exist" containerID="2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff"
Sep 16 13:34:00.615238 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.615182    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff"} err="failed to get container status \"2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff\": rpc error: code = NotFound desc = could not find container \"2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff\": container with ID starting with 2ec2422d1af9883480b02001e6aff72a2202bb7db355f6524621b558160adbff not found: ID does not exist"
Sep 16 13:34:00.615238 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.615210    2558 scope.go:117] "RemoveContainer" containerID="c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8"
Sep 16 13:34:00.615512 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:34:00.615495    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8\": container with ID starting with c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8 not found: ID does not exist" containerID="c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8"
Sep 16 13:34:00.615570 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.615520    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8"} err="failed to get container status \"c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8\": rpc error: code = NotFound desc = could not find container \"c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8\": container with ID starting with c65044a9d03ead5175b18ff21f4e20192bee2dbf665a80370b22916940dff4d8 not found: ID does not exist"
Sep 16 13:34:00.616574 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.616553    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"]
Sep 16 13:34:00.622376 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:00.622354    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11631-predictor-58dc68f8c4-spbkr"]
Sep 16 13:34:02.041952 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:02.041920    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" path="/var/lib/kubelet/pods/d06f5a66-69c5-4c5d-9419-d76fcf88b6e7/volumes"
Sep 16 13:34:04.720612 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:04.720574    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:34:09.720464 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:09.720431    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:34:09.720872 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:09.720530    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:34:14.720803 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:14.720764    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:34:19.720239 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:19.720149    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:34:24.720301 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:24.720258    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:34:26.856439 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:26.856414    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:34:26.893148 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:26.893112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls\") pod \"265defd5-6763-430d-8133-9261ea2bcfd8\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") "
Sep 16 13:34:26.893328 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:26.893191    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/265defd5-6763-430d-8133-9261ea2bcfd8-openshift-service-ca-bundle\") pod \"265defd5-6763-430d-8133-9261ea2bcfd8\" (UID: \"265defd5-6763-430d-8133-9261ea2bcfd8\") "
Sep 16 13:34:26.893565 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:26.893536    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265defd5-6763-430d-8133-9261ea2bcfd8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "265defd5-6763-430d-8133-9261ea2bcfd8" (UID: "265defd5-6763-430d-8133-9261ea2bcfd8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:34:26.895165 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:26.895137    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "265defd5-6763-430d-8133-9261ea2bcfd8" (UID: "265defd5-6763-430d-8133-9261ea2bcfd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:34:26.993954 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:26.993854    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/265defd5-6763-430d-8133-9261ea2bcfd8-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:34:26.993954 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:26.993899    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/265defd5-6763-430d-8133-9261ea2bcfd8-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:34:27.712233 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.712200    2558 generic.go:356] "Generic (PLEG): container finished" podID="265defd5-6763-430d-8133-9261ea2bcfd8" containerID="d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a" exitCode=0
Sep 16 13:34:27.712418 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.712241    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"
Sep 16 13:34:27.712418 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.712298    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" event={"ID":"265defd5-6763-430d-8133-9261ea2bcfd8","Type":"ContainerDied","Data":"d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a"}
Sep 16 13:34:27.712418 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.712351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp" event={"ID":"265defd5-6763-430d-8133-9261ea2bcfd8","Type":"ContainerDied","Data":"98217551c6dc930d7653a46c76a12d5b853fa9a2ff3d4cbfc3694420e3852aeb"}
Sep 16 13:34:27.712418 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.712373    2558 scope.go:117] "RemoveContainer" containerID="d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a"
Sep 16 13:34:27.720890 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.720727    2558 scope.go:117] "RemoveContainer" containerID="d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a"
Sep 16 13:34:27.721036 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:34:27.721016    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a\": container with ID starting with d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a not found: ID does not exist" containerID="d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a"
Sep 16 13:34:27.721083 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.721045    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a"} err="failed to get container status \"d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a\": rpc error: code = NotFound desc = could not find container \"d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a\": container with ID starting with d176e67dec890300b5aae1c10f36c8126ba203cbe5aa005a4373f4d20def029a not found: ID does not exist"
Sep 16 13:34:27.734305 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.734276    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"]
Sep 16 13:34:27.737952 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:27.737931    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11631-6b777c6c96-4j7bp"]
Sep 16 13:34:28.041573 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:34:28.041475    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" path="/var/lib/kubelet/pods/265defd5-6763-430d-8133-9261ea2bcfd8/volumes"
Sep 16 13:41:50.912177 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:50.912095    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"]
Sep 16 13:41:50.914575 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:50.912858    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820" containerID="cri-o://44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863" gracePeriod=30
Sep 16 13:41:51.065756 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:51.065720    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"]
Sep 16 13:41:51.066017 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:51.065967    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container" containerID="cri-o://fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e" gracePeriod=30
Sep 16 13:41:51.066117 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:51.065991    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kube-rbac-proxy" containerID="cri-o://f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c" gracePeriod=30
Sep 16 13:41:51.311592 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:51.311548    2558 generic.go:356] "Generic (PLEG): container finished" podID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerID="f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c" exitCode=2
Sep 16 13:41:51.311763 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:51.311613    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" event={"ID":"a51c368d-7fb5-4c59-905f-784565b4d1f9","Type":"ContainerDied","Data":"f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"}
Sep 16 13:41:53.902430 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:53.902406    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:41:53.974728 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:53.974693    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls\") pod \"a51c368d-7fb5-4c59-905f-784565b4d1f9\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") "
Sep 16 13:41:53.974932 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:53.974768    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-rbac-proxy-sar-config\") pod \"a51c368d-7fb5-4c59-905f-784565b4d1f9\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") "
Sep 16 13:41:53.974932 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:53.974872    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8krtl\" (UniqueName: \"kubernetes.io/projected/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-api-access-8krtl\") pod \"a51c368d-7fb5-4c59-905f-784565b4d1f9\" (UID: \"a51c368d-7fb5-4c59-905f-784565b4d1f9\") "
Sep 16 13:41:53.975146 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:53.975124    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a51c368d-7fb5-4c59-905f-784565b4d1f9" (UID: "a51c368d-7fb5-4c59-905f-784565b4d1f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:41:53.976833 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:53.976790    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a51c368d-7fb5-4c59-905f-784565b4d1f9" (UID: "a51c368d-7fb5-4c59-905f-784565b4d1f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:41:53.976945 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:53.976861    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-api-access-8krtl" (OuterVolumeSpecName: "kube-api-access-8krtl") pod "a51c368d-7fb5-4c59-905f-784565b4d1f9" (UID: "a51c368d-7fb5-4c59-905f-784565b4d1f9"). InnerVolumeSpecName "kube-api-access-8krtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:41:54.076122 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.076087    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:41:54.076122 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.076117    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8krtl\" (UniqueName: \"kubernetes.io/projected/a51c368d-7fb5-4c59-905f-784565b4d1f9-kube-api-access-8krtl\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:41:54.076122 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.076126    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a51c368d-7fb5-4c59-905f-784565b4d1f9-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:41:54.324754 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.324659    2558 generic.go:356] "Generic (PLEG): container finished" podID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerID="fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e" exitCode=0
Sep 16 13:41:54.324754 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.324696    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" event={"ID":"a51c368d-7fb5-4c59-905f-784565b4d1f9","Type":"ContainerDied","Data":"fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"}
Sep 16 13:41:54.324754 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.324719    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"
Sep 16 13:41:54.324754 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.324741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl" event={"ID":"a51c368d-7fb5-4c59-905f-784565b4d1f9","Type":"ContainerDied","Data":"39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04"}
Sep 16 13:41:54.325105 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.324767    2558 scope.go:117] "RemoveContainer" containerID="f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"
Sep 16 13:41:54.331837 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.331797    2558 scope.go:117] "RemoveContainer" containerID="f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"
Sep 16 13:41:54.333134 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.333038    2558 scope.go:117] "RemoveContainer" containerID="fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"
Sep 16 13:41:54.335269 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:41:54.335243    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_kube-rbac-proxy_error-404-isvc-a1820-predictor-79bf8db794-hl7vl_kserve-ci-e2e-test_a51c368d-7fb5-4c59-905f-784565b4d1f9_0 in pod sandbox 39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04 from index: no such id: 'f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c'" containerID="f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"
Sep 16 13:41:54.335387 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:41:54.335285    2558 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_kube-rbac-proxy_error-404-isvc-a1820-predictor-79bf8db794-hl7vl_kserve-ci-e2e-test_a51c368d-7fb5-4c59-905f-784565b4d1f9_0 in pod sandbox 39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04 from index: no such id: 'f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c'" containerID="f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"
Sep 16 13:41:54.335387 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.335326    2558 scope.go:117] "RemoveContainer" containerID="fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"
Sep 16 13:41:54.340616 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:41:54.340590    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_kserve-container_error-404-isvc-a1820-predictor-79bf8db794-hl7vl_kserve-ci-e2e-test_a51c368d-7fb5-4c59-905f-784565b4d1f9_0 in pod sandbox 39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04 from index: no such id: 'fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e'" containerID="fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"
Sep 16 13:41:54.340712 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.340621    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"} err="rpc error: code = Unknown desc = failed to delete container k8s_kserve-container_error-404-isvc-a1820-predictor-79bf8db794-hl7vl_kserve-ci-e2e-test_a51c368d-7fb5-4c59-905f-784565b4d1f9_0 in pod sandbox 39d358654e50d4f6361eedabfcb2cbf670c379d4dcce857116a52b0c090d7d04 from index: no such id: 'fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e'"
Sep 16 13:41:54.340712 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.340638    2558 scope.go:117] "RemoveContainer" containerID="f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"
Sep 16 13:41:54.340914 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:41:54.340896    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c\": container with ID starting with f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c not found: ID does not exist" containerID="f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"
Sep 16 13:41:54.340981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.340924    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c"} err="failed to get container status \"f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c\": rpc error: code = NotFound desc = could not find container \"f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c\": container with ID starting with f7e27edd6f31e6f7921966f920678e4d355351af1929bb84ec7fe736ce55336c not found: ID does not exist"
Sep 16 13:41:54.340981 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.340940    2558 scope.go:117] "RemoveContainer" containerID="fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"
Sep 16 13:41:54.341195 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:41:54.341163    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e\": container with ID starting with fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e not found: ID does not exist" containerID="fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"
Sep 16 13:41:54.341248 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.341195    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e"} err="failed to get container status \"fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e\": rpc error: code = NotFound desc = could not find container \"fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e\": container with ID starting with fbe622d295b7f7a9fe53e0ebfecbc33903396961830bbd2ae14593585d23310e not found: ID does not exist"
Sep 16 13:41:54.344543 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.344521    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"]
Sep 16 13:41:54.351050 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.351030    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1820-predictor-79bf8db794-hl7vl"]
Sep 16 13:41:54.679209 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:54.679171    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:41:56.041462 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:56.041431    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" path="/var/lib/kubelet/pods/a51c368d-7fb5-4c59-905f-784565b4d1f9/volumes"
Sep 16 13:41:59.678939 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:41:59.678896    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:42:04.679306 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:04.679263    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:42:04.679712 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:04.679376    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:42:07.733799 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:07.733762    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:07.735578 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:07.735546    2558 ???:1] "http: TLS handshake error from 10.0.141.139:41710: EOF"
Sep 16 13:42:08.550586 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:08.550553    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:09.372730 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:09.372690    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:09.679077 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:09.679041    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:42:10.153084 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:10.153055    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:10.935443 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:10.935403    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:11.721764 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:11.721738    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:12.504674 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:12.504638    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:13.278918 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:13.278884    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:14.070753 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:14.070713    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:14.679322 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:14.679277    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:42:14.859077 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:14.859044    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:15.642798 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:15.642762    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:16.432570 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:16.432534    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:17.233905 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:17.233876    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:18.056721 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:18.056689    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a1820-864897f7c9-6pkdx_40e56f7c-0ced-489f-8b30-182c21fd80a2/switch-graph-a1820/0.log"
Sep 16 13:42:19.679243 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:19.679208    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:42:21.056104 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.056076    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:42:21.190918 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.190830    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40e56f7c-0ced-489f-8b30-182c21fd80a2-openshift-service-ca-bundle\") pod \"40e56f7c-0ced-489f-8b30-182c21fd80a2\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") "
Sep 16 13:42:21.191080 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.190951    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls\") pod \"40e56f7c-0ced-489f-8b30-182c21fd80a2\" (UID: \"40e56f7c-0ced-489f-8b30-182c21fd80a2\") "
Sep 16 13:42:21.191218 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.191200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e56f7c-0ced-489f-8b30-182c21fd80a2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "40e56f7c-0ced-489f-8b30-182c21fd80a2" (UID: "40e56f7c-0ced-489f-8b30-182c21fd80a2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:42:21.193005 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.192981    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40e56f7c-0ced-489f-8b30-182c21fd80a2" (UID: "40e56f7c-0ced-489f-8b30-182c21fd80a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:42:21.291682 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.291627    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e56f7c-0ced-489f-8b30-182c21fd80a2-proxy-tls\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:42:21.291682 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.291677    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40e56f7c-0ced-489f-8b30-182c21fd80a2-openshift-service-ca-bundle\") on node \"ip-10-0-139-96.ec2.internal\" DevicePath \"\""
Sep 16 13:42:21.419997 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.419963    2558 generic.go:356] "Generic (PLEG): container finished" podID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerID="44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863" exitCode=0
Sep 16 13:42:21.420168 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.420007    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"
Sep 16 13:42:21.420168 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.420006    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" event={"ID":"40e56f7c-0ced-489f-8b30-182c21fd80a2","Type":"ContainerDied","Data":"44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863"}
Sep 16 13:42:21.420168 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.420071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx" event={"ID":"40e56f7c-0ced-489f-8b30-182c21fd80a2","Type":"ContainerDied","Data":"853c43e458129eb87aec9234006164e418025d7a59241e69d2f3f411208e5b3c"}
Sep 16 13:42:21.420168 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.420102    2558 scope.go:117] "RemoveContainer" containerID="44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863"
Sep 16 13:42:21.428783 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.428765    2558 scope.go:117] "RemoveContainer" containerID="44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863"
Sep 16 13:42:21.429097 ip-10-0-139-96 kubenswrapper[2558]: E0916 13:42:21.429077    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863\": container with ID starting with 44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863 not found: ID does not exist" containerID="44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863"
Sep 16 13:42:21.429166 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.429104    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863"} err="failed to get container status \"44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863\": rpc error: code = NotFound desc = could not find container \"44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863\": container with ID starting with 44ff278ab32ca5bac70c6e9e160fe99c1e1c423e66e22f2f8c1dda536091c863 not found: ID does not exist"
Sep 16 13:42:21.442545 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.442482    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"]
Sep 16 13:42:21.445450 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:21.445427    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a1820-864897f7c9-6pkdx"]
Sep 16 13:42:22.041953 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:22.041919    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" path="/var/lib/kubelet/pods/40e56f7c-0ced-489f-8b30-182c21fd80a2/volumes"
Sep 16 13:42:23.676493 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676456    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-svrn2/must-gather-z2s8d"]
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676735    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676747    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676761    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kube-rbac-proxy"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676767    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kube-rbac-proxy"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676773    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676778    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676787    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676795    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676805    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kube-rbac-proxy"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676827    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kube-rbac-proxy"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676846    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676851    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820"
Sep 16 13:42:23.676900 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676914    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kube-rbac-proxy"
Sep 16 13:42:23.677319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676922    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="265defd5-6763-430d-8133-9261ea2bcfd8" containerName="splitter-graph-11631"
Sep 16 13:42:23.677319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676930    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51c368d-7fb5-4c59-905f-784565b4d1f9" containerName="kserve-container"
Sep 16 13:42:23.677319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676936    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kserve-container"
Sep 16 13:42:23.677319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676941    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06f5a66-69c5-4c5d-9419-d76fcf88b6e7" containerName="kube-rbac-proxy"
Sep 16 13:42:23.677319 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.676946    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e56f7c-0ced-489f-8b30-182c21fd80a2" containerName="switch-graph-a1820"
Sep 16 13:42:23.680982 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.680963    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:23.683551 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.683531    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-svrn2\"/\"openshift-service-ca.crt\""
Sep 16 13:42:23.684767 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.684748    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-svrn2\"/\"kube-root-ca.crt\""
Sep 16 13:42:23.684863 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.684748    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-svrn2\"/\"default-dockercfg-prqsv\""
Sep 16 13:42:23.694273 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.688527    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svrn2/must-gather-z2s8d"]
Sep 16 13:42:23.811682 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.811643    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw989\" (UniqueName: \"kubernetes.io/projected/eff66312-dc5d-4a36-90a1-776b9f41e5fe-kube-api-access-rw989\") pod \"must-gather-z2s8d\" (UID: \"eff66312-dc5d-4a36-90a1-776b9f41e5fe\") " pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:23.811893 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.811722    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eff66312-dc5d-4a36-90a1-776b9f41e5fe-must-gather-output\") pod \"must-gather-z2s8d\" (UID: \"eff66312-dc5d-4a36-90a1-776b9f41e5fe\") " pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:23.912450 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.912401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eff66312-dc5d-4a36-90a1-776b9f41e5fe-must-gather-output\") pod \"must-gather-z2s8d\" (UID: \"eff66312-dc5d-4a36-90a1-776b9f41e5fe\") " pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:23.912635 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.912462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw989\" (UniqueName: \"kubernetes.io/projected/eff66312-dc5d-4a36-90a1-776b9f41e5fe-kube-api-access-rw989\") pod \"must-gather-z2s8d\" (UID: \"eff66312-dc5d-4a36-90a1-776b9f41e5fe\") " pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:23.912747 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.912725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eff66312-dc5d-4a36-90a1-776b9f41e5fe-must-gather-output\") pod \"must-gather-z2s8d\" (UID: \"eff66312-dc5d-4a36-90a1-776b9f41e5fe\") " pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:23.923509 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.923483    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw989\" (UniqueName: \"kubernetes.io/projected/eff66312-dc5d-4a36-90a1-776b9f41e5fe-kube-api-access-rw989\") pod \"must-gather-z2s8d\" (UID: \"eff66312-dc5d-4a36-90a1-776b9f41e5fe\") " pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:23.990648 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:23.990570    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svrn2/must-gather-z2s8d"
Sep 16 13:42:24.270569 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:24.270494    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svrn2/must-gather-z2s8d"]
Sep 16 13:42:24.275116 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:42:24.275089    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff66312_dc5d_4a36_90a1_776b9f41e5fe.slice/crio-da382ea6cc3125bce99ea777c9f873cedec35a6d8f05076dc771c4791bd2f517 WatchSource:0}: Error finding container da382ea6cc3125bce99ea777c9f873cedec35a6d8f05076dc771c4791bd2f517: Status 404 returned error can't find the container with id da382ea6cc3125bce99ea777c9f873cedec35a6d8f05076dc771c4791bd2f517
Sep 16 13:42:24.276913 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:24.276895    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:42:24.433144 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:24.433111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svrn2/must-gather-z2s8d" event={"ID":"eff66312-dc5d-4a36-90a1-776b9f41e5fe","Type":"ContainerStarted","Data":"da382ea6cc3125bce99ea777c9f873cedec35a6d8f05076dc771c4791bd2f517"}
Sep 16 13:42:25.440002 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:25.439962    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svrn2/must-gather-z2s8d" event={"ID":"eff66312-dc5d-4a36-90a1-776b9f41e5fe","Type":"ContainerStarted","Data":"6c31533f23ad70ffe269d9a1c6ab3ad4a33ebf702bff2848d0cf49c8a6c35c7d"}
Sep 16 13:42:25.440497 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:25.440021    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svrn2/must-gather-z2s8d" event={"ID":"eff66312-dc5d-4a36-90a1-776b9f41e5fe","Type":"ContainerStarted","Data":"815d00bb84cdd63dbd29a764634b3a9a2d6db26af41fc6ff6119d51d4231c272"}
Sep 16 13:42:25.460879 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:25.460798    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-svrn2/must-gather-z2s8d" podStartSLOduration=1.841562771 podStartE2EDuration="2.46077604s" totalImagesPullingTime="619.213269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:42:25.459114871 +0000 UTC m=+3271.920743566" watchObservedRunningTime="2026-09-16 13:42:25.46077604 +0000 UTC m=+3271.922404932"
Sep 16 13:42:26.306101 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:26.306065    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2mddz_8a48d3d0-74d6-4512-8a11-e79ca407a8e4/global-pull-secret-syncer/0.log"
Sep 16 13:42:26.465626 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:26.465594    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-njp7q_c370ba8b-56d2-46e1-8ed2-baae87e9df49/konnectivity-agent/0.log"
Sep 16 13:42:26.542036 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:26.542008    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-96.ec2.internal_b846e37f4ef604879d01c53444ca7848/haproxy/0.log"
Sep 16 13:42:30.818488 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:30.818413    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55b45f6cbd-7nzv8_7e5f4771-0598-4865-9a90-299c9cbcd895/metrics-server/0.log"
Sep 16 13:42:30.841585 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:30.841552    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-hvcsv_4382446e-1349-4594-bf2e-e2633012b722/monitoring-plugin/0.log"
Sep 16 13:42:30.869656 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:30.869630    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgjwx_f8cac50e-c94f-4460-902c-5c2e7e3c075d/node-exporter/0.log"
Sep 16 13:42:30.903872 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:30.903783    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgjwx_f8cac50e-c94f-4460-902c-5c2e7e3c075d/kube-rbac-proxy/0.log"
Sep 16 13:42:30.952921 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:30.952884    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgjwx_f8cac50e-c94f-4460-902c-5c2e7e3c075d/init-textfile/0.log"
Sep 16 13:42:31.507220 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:31.507172    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-zkpf8_18132788-212f-4427-a048-cce30d147a6d/prometheus-operator/0.log"
Sep 16 13:42:31.521195 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:31.521166    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-zkpf8_18132788-212f-4427-a048-cce30d147a6d/kube-rbac-proxy/0.log"
Sep 16 13:42:33.875379 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:33.875343    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c8f769787-2dkcv_00e9118b-e1b1-44c8-b4ab-94d95cc8f8e5/console/0.log"
Sep 16 13:42:33.914344 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:33.914315    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-c9hsx_19f33b29-948e-4d9c-bb45-1e036c46358f/download-server/0.log"
Sep 16 13:42:34.515471 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.515441    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"]
Sep 16 13:42:34.521430 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.521404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.528151 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.528124    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"]
Sep 16 13:42:34.613166 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.613126    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcz8\" (UniqueName: \"kubernetes.io/projected/04ee563c-f37b-45af-90cb-86380f5ac681-kube-api-access-mxcz8\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.613493 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.613462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-podres\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.613730 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.613708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-lib-modules\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.614008 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.613987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-proc\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.614146 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.614128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-sys\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.715293 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-podres\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.715650 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715619    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-podres\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.715776 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-lib-modules\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.715776 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-proc\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.715776 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-sys\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.715776 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-lib-modules\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.715776 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715769    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-proc\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.716095 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcz8\" (UniqueName: \"kubernetes.io/projected/04ee563c-f37b-45af-90cb-86380f5ac681-kube-api-access-mxcz8\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.716095 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.715844    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04ee563c-f37b-45af-90cb-86380f5ac681-sys\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.724854 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.724800    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcz8\" (UniqueName: \"kubernetes.io/projected/04ee563c-f37b-45af-90cb-86380f5ac681-kube-api-access-mxcz8\") pod \"perf-node-gather-daemonset-s6qwk\" (UID: \"04ee563c-f37b-45af-90cb-86380f5ac681\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:34.832875 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:34.832784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:35.121902 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.121719    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"]
Sep 16 13:42:35.125052 ip-10-0-139-96 kubenswrapper[2558]: W0916 13:42:35.125015    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04ee563c_f37b_45af_90cb_86380f5ac681.slice/crio-96e4086618c29741c01b9ab382526126ddfda4fb331fb1a7775c8deb3ce14e27 WatchSource:0}: Error finding container 96e4086618c29741c01b9ab382526126ddfda4fb331fb1a7775c8deb3ce14e27: Status 404 returned error can't find the container with id 96e4086618c29741c01b9ab382526126ddfda4fb331fb1a7775c8deb3ce14e27
Sep 16 13:42:35.367231 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.367186    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gc2qg_96043e4b-3047-471a-909f-4d1e0769cc57/dns/0.log"
Sep 16 13:42:35.386855 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.386768    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gc2qg_96043e4b-3047-471a-909f-4d1e0769cc57/kube-rbac-proxy/0.log"
Sep 16 13:42:35.462938 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.462911    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d2szc_839b45b7-c79f-440d-ac6b-6390e71c58b6/dns-node-resolver/0.log"
Sep 16 13:42:35.494234 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.494202    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk" event={"ID":"04ee563c-f37b-45af-90cb-86380f5ac681","Type":"ContainerStarted","Data":"0e162ffc2e6aaf650a403917a59eb1663156be99020d085858b686cb16927680"}
Sep 16 13:42:35.494430 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.494280    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk" event={"ID":"04ee563c-f37b-45af-90cb-86380f5ac681","Type":"ContainerStarted","Data":"96e4086618c29741c01b9ab382526126ddfda4fb331fb1a7775c8deb3ce14e27"}
Sep 16 13:42:35.494430 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.494312    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:35.512252 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:35.512197    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk" podStartSLOduration=1.512179947 podStartE2EDuration="1.512179947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:42:35.511649545 +0000 UTC m=+3281.973278259" watchObservedRunningTime="2026-09-16 13:42:35.512179947 +0000 UTC m=+3281.973808650"
Sep 16 13:42:36.014617 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:36.014582    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mqv9c_e23acb97-072a-4d73-9411-6aee7d0b9259/node-ca/0.log"
Sep 16 13:42:36.743783 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:36.743753    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbbdd75d-mdklm_f9e9ee04-dd2e-492e-9a74-1ef0d951cbff/router/0.log"
Sep 16 13:42:37.079568 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:37.079486    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n4xkz_c0f3be5c-61bb-4f1a-be5f-18484d33e0f4/serve-healthcheck-canary/0.log"
Sep 16 13:42:37.555133 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:37.555102    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-np6cj_3d8b0d29-3267-449c-a4f7-784bfe0113fb/kube-rbac-proxy/0.log"
Sep 16 13:42:37.570320 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:37.570296    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-np6cj_3d8b0d29-3267-449c-a4f7-784bfe0113fb/exporter/0.log"
Sep 16 13:42:37.588623 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:37.588600    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-np6cj_3d8b0d29-3267-449c-a4f7-784bfe0113fb/extractor/0.log"
Sep 16 13:42:39.854368 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:39.854340    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-7jj7p_044a3d0f-607e-4ff9-8c65-83fc6b71fd17/server/0.log"
Sep 16 13:42:40.001905 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:40.001869    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-nk86t_f813514f-6a80-45b0-84d3-c8d72fa03762/manager/0.log"
Sep 16 13:42:41.508040 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:41.508011    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-s6qwk"
Sep 16 13:42:45.551420 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.551388    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tt472_93857f18-85a0-4668-87ab-8c83030b026c/kube-multus-additional-cni-plugins/0.log"
Sep 16 13:42:45.573267 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.573239    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tt472_93857f18-85a0-4668-87ab-8c83030b026c/egress-router-binary-copy/0.log"
Sep 16 13:42:45.598312 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.598281    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tt472_93857f18-85a0-4668-87ab-8c83030b026c/cni-plugins/0.log"
Sep 16 13:42:45.617636 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.617609    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tt472_93857f18-85a0-4668-87ab-8c83030b026c/bond-cni-plugin/0.log"
Sep 16 13:42:45.635888 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.635863    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tt472_93857f18-85a0-4668-87ab-8c83030b026c/routeoverride-cni/0.log"
Sep 16 13:42:45.655246 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.655222    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tt472_93857f18-85a0-4668-87ab-8c83030b026c/whereabouts-cni-bincopy/0.log"
Sep 16 13:42:45.672239 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.672209    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tt472_93857f18-85a0-4668-87ab-8c83030b026c/whereabouts-cni/0.log"
Sep 16 13:42:45.739027 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.738995    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nn9ht_13252a94-a763-48d6-b1d8-6a4bb134ec77/kube-multus/0.log"
Sep 16 13:42:45.788514 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.788432    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cblhw_9d28d3da-9cae-48ec-ac36-003b8aa41bb6/network-metrics-daemon/0.log"
Sep 16 13:42:45.802129 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:45.802090    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cblhw_9d28d3da-9cae-48ec-ac36-003b8aa41bb6/kube-rbac-proxy/0.log"
Sep 16 13:42:46.739762 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.739735    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/ovn-controller/0.log"
Sep 16 13:42:46.753099 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.753071    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/ovn-acl-logging/0.log"
Sep 16 13:42:46.773118 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.773090    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/kube-rbac-proxy-node/0.log"
Sep 16 13:42:46.790916 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.790887    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 13:42:46.806346 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.806316    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/northd/0.log"
Sep 16 13:42:46.822991 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.822971    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/nbdb/0.log"
Sep 16 13:42:46.841293 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.841271    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/sbdb/0.log"
Sep 16 13:42:46.901321 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:46.901290    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wb288_551e18e6-7f02-42aa-8b5f-2132d9df8fcd/ovnkube-controller/0.log"
Sep 16 13:42:48.195744 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:48.195713    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qs6rn_3b346991-b5d0-4414-b808-6380a553c4a5/network-check-target-container/0.log"
Sep 16 13:42:49.103519 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:49.103490    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7qrml_be79be6c-655b-4a79-be76-7fb70374c8c6/iptables-alerter/0.log"
Sep 16 13:42:49.686357 ip-10-0-139-96 kubenswrapper[2558]: I0916 13:42:49.686329    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-f2f9h_106cce76-3ca1-4e9e-a859-0e0ad9f9e37a/tuned/0.log"